Your AI System Has Technical Debt. It’s Just Not Where You’re Looking.

Technical debt used to be relatively easy to recognize.

Messy code, duplicated logic, an abstraction that should never have survived the prototype, a database column called status2 that everyone is afraid to rename.

You might not fix it. You might build an entire career around carefully avoiding it. But at least you know where it lives.

AI has made this more interesting.

I've been building an assistant for my website recently. Its job isn't particularly exotic: understand why someone came, answer useful questions, and if there's a good reason for us to talk, help make that happen.

At one point it did exactly that. It had a good conversation, understood the visitor, correctly figured out that we should talk, and convinced them to get in touch.

Then it forgot to collect their contact details.

The AI succeeded.

The product failed.

Which is a fairly good summary of the industry right now.

The bug wasn't really in the code. It was somewhere in the increasingly weird layer we're all building around models: prompts, instructions, examples, retrieved documents, tool descriptions, business rules and assumptions.

We still tend to treat prompts as something halfway between configuration and copywriting. Maybe because calling them "prompts" makes them sound harmless. Like something you type before asking a computer to draw a cat wearing sunglasses.

But if changing three sentences can change how your product makes decisions, those sentences are part of your production system.

And congratulations: they can accumulate technical debt too.

The process is beautifully familiar.

A conversation fails, so you add an instruction. Another one fails, so you add an exception. Then an example. Then a document. Then another instruction explaining how to interpret the first instruction.

Eventually you have 4,000 words of carefully evolved context and nobody is entirely sure which 600 words are keeping the whole thing alive.

We have invented global mutable state in English.

Something changes over here, something unrelated behaves differently over there, nobody knows why, and eventually the team settles on that ancient and highly respected engineering principle:

Don't fucking touch it.

The problem gets stranger because context isn't just prompts.

An agent might see documentation, previous messages, retrieved knowledge, tool descriptions and dynamically generated state. Give it more information and we instinctively think we've made it smarter.

Sometimes we have.

Sometimes we've handed it two contradictory documents and enough intelligence to confidently choose the wrong one.

This is where AI technical debt starts looking a lot like organizational debt.

Imagine sales believes one thing about the product, engineering another, the documentation says something slightly different, and the actual answer exists in a Slack message from three weeks ago between two people who are currently on vacation.

Humans are surprisingly good at surviving this. We ask around. We remember who actually knows. We develop organizational folklore. Every company eventually acquires at least one person who functions as an undocumented distributed database.

Then we connect an agent to everything.

Congratulations. We've automated the confusion.

RAG doesn't create truth. More context doesn't create clarity. Sometimes the most useful thing an AI system does is expose disagreements that humans had quietly learned to work around.

Building my assistant has changed how I think about this.

I found myself writing rules like:

Ask a discovery question only if the answer would materially change whether I should talk to this person.

At some point I looked at that sentence and thought: hang on.

This isn't really a prompt anymore.

It's a business rule written in natural language.

We spent decades putting business logic into increasingly sophisticated programming languages, and now we're putting some of it back into English.

Apparently we missed meetings.

And business rules need tests.

Not just "did the AI give a good answer?" but: did it ask an unnecessary question? Did it invent something? Did it recognize when someone wanted to talk? Did it collect what was needed for the handoff?

This matters even more with agents, because their mistakes don't necessarily end as bad sentences. They can become actions.

Better models won't remove this problem either.

A more capable agent can follow a bad instruction more effectively, travel much further with a wrong assumption, use several tools along the way, and then produce a beautifully formatted explanation of why everything went according to plan.

We used to call this a bug.

Now it has bullet points.

So I'm starting to treat the stuff around the model more like software.

Important prompts get versioned. Important behaviors get tested. Context needs ownership. Model upgrades deserve regression tests. Agents need boundaries around what they can read, decide and modify.

None of this is particularly revolutionary.

That's the funny part.

For all the talk about how AI changes software engineering forever, quite a lot of the job seems to involve rediscovering lessons software engineering already learned, giving them new names, and putting them into conference talks.

The code may now be the cleanest part of the system.

Somewhere, a team is staring at a 6,000-word system prompt containing three years of accumulated wisdom, exceptions, experiments and emergency fixes.

Nobody knows exactly why it works.

Nobody remembers who wrote half of it.

Nobody wants to change it.

So eventually someone will put a comment above it:

// DO NOT TOUCH

And just like that, the future has arrived.