The model will tell you it worked when it didn’t

Claude Code finishes something and tells you what it did, and that report is the most trustworthy-sounding part of the whole exchange, because it arrives in the past tense from something that was there. The report is produced the same way as everything else, which makes it a plausible account of what probably happened, assembled afterwards.

Counting attempts and calling them results

This site has a small feature that notifies search engines when a page changes. It ran, told me how many pages it had submitted, and the number it gave me was the number it had tried to submit. The service was rejecting all of them. The count was counting attempts and presenting them as results, and I only found out by going looking for something else.

The same week, a control for clearing cached pages reported success for page names that didn’t match anything real. It had done what it was asked, which was to issue the instruction, and it reported on having issued it rather than on whether it had landed (which is what actually matters).

Reading back through my notes for that quarter, 3 separate features had shipped with the same defect: a success report that didn’t reflect what had actually been verified. I didn’t spot the pattern while it was happening, only later, once the notes were in one place.

“The tests pass” belongs to the same family, since it’s a true sentence about the tests. On my food journal, a safety check got added that made one operation 22 times slower, and the whole suite stayed green, because it was checking whether results were correct rather than how long they took. One test happened to have a time limit on it, which is the only reason it surfaced.

Why it’s hard to catch

A false success report costs nothing to produce and reads like a true one, so the usual defense of noticing that something seems off doesn’t fire. There’s no hedging in it, no vagueness to catch on, and it’s often correct, which is what makes the habit of skimming it so easy to acquire.

The failures also tend to be quiet by construction. A crash announces itself, but a job that reports a pile of successful submissions but actually made none looks precisely like a job that worked, and it will go on looking like that indefinitely.

What I do now

Ask for the evidence rather than the conclusion. Go and look at the thing itself, in the place it actually lives. And say what “done” means before the work starts, because otherwise the definition gets written afterwards by the same process that decides whether it was met, so it will present you with a decision based on the work, instead of basing the work on an expected result.

None of it is clever, and it’s slow: this is the area where I gave back most of the time I’d saved, and I haven’t found a way around it that I trust. I still skim reports, usually late in a session when everything has been going well, which is the worst moment to start assuming things are fine.

The next post is about the version of this that survives every check, because everything genuinely worked.

If you have a check that catches these earlier than mine do, I’d like to steal it (@roland.leth.ro / @rolandleth).

The rest of the series is listed in What changed my mind about building with AI.