The Bug That Taught Me About AI Cost Governance

The bug was embarrassingly ordinary. A feature I built for one of my own tools was supposed to classify a stream of incoming activity and flag what actually needed a response. It classified everything, every time the page loaded, and never saved the result. So every refresh re-ran the same paid model calls from scratch.

In a single day it burned through millions of tokens, tripped my rate limit, and started knocking out other features that happened to share the same API key. What actually stopped it was a hard spend cap I'd set months earlier, more out of habit than foresight.

The easy fix was obvious: save the result so the work doesn't get redone. That change took about ten minutes. The real work started after, in parts of the system that had nothing to do with the bug itself.

First, I split the one shared key into separate keys per workload, so a runaway in any single place can no longer starve everything else. Then I put a hard daily ceiling on each of those keys, a limit that stops the workload cold the moment it's crossed, not a dashboard alert someone has to notice. Last, and most important, I added a test that fails the build if any code path spends money on a model call without persisting what it gets back. That test is the actual prevention. The spend cap only ever told me the fire was already burning.

That's the part I keep coming back to when I talk with leaders about AI governance. Everyone reaches for the spend cap first, because it's visible and easy to point to in a slide. A cap behaves like a smoke alarm: it tells you something's already wrong, after the fact. The sprinkler system is the boring, unglamorous stuff — isolated keys so one failure can't take down the rest, hard ceilings instead of soft alerts, and a test that makes the dangerous pattern impossible to ship twice.

None of this required a framework or a vendor tool. It required treating a personal, low-stakes bug as seriously as I'd want a client's production incident treated, because the failure mode is identical either way — only the size of the bill changes.

So the question worth sitting with: if something you're running went sideways tonight, would anything actually stop the spend before morning, or would you find out when the invoice arrives?