All notes

Honest work refuses to invent.

The seductive failure mode of an AI agent is fluent confidence. The structural fix is a design property — and it has to hold at more than one layer.

Sky · · 4 min read

Most operators have a story about the moment they stopped trusting an AI agent. The agent confidently produced a phone number that wasn't real. A citation that didn't exist. A competitor analysis that turned out to be three plausible-sounding paragraphs of training data with the word the in the right places. The output read like work. It wasn't.

That's the fluent failure mode. It is, structurally, the worst mode an AI agent can fail in — because it is indistinguishable from competence. The work that's wrong looks the same as the work that's right. The operator finds out the difference only after the fact, when something downstream of the fluent answer hits reality and breaks.

The fluent failure mode is the worst failure mode. It is indistinguishable from competence.

The temptation that produces it is built into the model. When a tool call fails or returns nothing, the language model underneath the agent still has its weights. It can produce something that fits the shape of the requested output — smooth, confident, on-topic. Most observers, watching an agent navigate a tool failure, instinctively reach for the phrase graceful degradation. That instinct is wrong. Graceful degradation is the bug, not the feature. The agent gracefully degrading is the agent making things up.

Most observers' instinct is to call training-data fill-in "graceful degradation." That instinct is wrong.

There are two kinds of agent. One that, when a tool call fails, fills the gap from training data and ships fluent prose. One that, when a tool call fails, acknowledges the gap, admits low confidence, and produces a partial report from only what it actually observed. The first looks better in a demo. The second is the only one that compounds in production.

This is the design choice, and it is upstream of the model. Honest work refuses to invent. The honest agent isn't a constrained agent — it's a correctly designed one. Its job is to report from a record of what actually happened, not to fill blank space with plausibility. Refusing to invent is not a limitation it tolerates. It is the only valid output for the work it's doing.

Honest work refuses to invent. The agent's job is to report from a record of what happened, not to fill blank space with plausibility.

The interesting part is what makes refusal possible. Refusal needs something to refuse from — a record of what was observed, separate from what could have been guessed. An agent without a substrate underneath has no refusal mechanism; it only has training data to fall back on. An agent with a substrate has something to point to, and something to honestly admit it doesn't have. The agent that brought the documents — and refused to make any up — is reading from the documents, not paraphrasing what similar documents tend to say.

That's the agent's job. The system's job is different. The system has to refuse to take the agent's honest blank and dress it up as completion. An agent can correctly admit a gap — and a pipeline downstream of that agent can still wrap the admission in a polished header, push it to a deliverable channel, and ship it to the customer as if the work succeeded. Honesty at the agent layer leaks at the next stage unless that stage is also designed to refuse. The customer ends up looking at a card marked Completed with an empty body. That's a different bug, with the same root: the system serving fabrication, just one layer up. The agent admits the gap. The system refuses to dress the admission up as completion. Both jobs are required.

This is also the unspoken prerequisite under last week's note. A workforce that funds its next turn out of last quarter's outcomes only works if last quarter's outcomes are real. A workforce that fabricates one quarter's work and then funds the next turn out of the fabricated success is a workforce compounding in the wrong direction. Outcomes pay the next turn — honest outcomes. The economics of the workforce era and the design of the workforce era are not separate conversations. The first depends entirely on the second.

A workforce that produces partial reports when it doesn't know is a workforce the customer learns to trust. A workforce that produces fluent reports for everything is one the customer eventually learns to ignore.

A workforce that produces partial reports when it doesn't know is one the customer learns to trust. A workforce that produces fluent reports for everything is one the customer eventually learns to ignore.

The category will divide on this line. It is already dividing.

The agent that brought the documents — and refused to make any up — is the only agent worth running. Refusal is the feature.