When the logs lie
A scenario I was generating kept coming out half-broken: some lines were proper dialogue, others reverted to placeholder text. It looked like the AI was producing bad output. It was not. The generation was clean every single time. Something was wiping the good result the instant it was written.
Three wrong theories
I went through three confident explanations before the right one: the output quality, then competing background workers, then a database quirk. Each was plausible. Each was wrong. What got me unstuck was a simple discipline: prove the theory with evidence before believing it.
The application logs were hiding it
The application’s own logs showed almost nothing during the failing window, which made it look like nothing was happening. That was the trap. Application logs lie by omission: filtered levels, missing pieces, whole categories of request that never get written down.
The truth was one layer down. The web server’s access log showed 150 save requests in sixty seconds during a single generation. A query against the database’s own row history showed hundreds of rows deleted inside that same minute. Suddenly it was obvious: the page’s auto-save and the background job were fighting. The job wrote good dialogue, the auto-save immediately overwrote it with the placeholder the page still had loaded, the next refresh reloaded the placeholder, and round it went.
What I took from it
When a request seems to be missing, do not trust the application log. The web server’s access log and the database’s own history are the ground truth, and they do not lie by omission the way a tidy application log does.
There was a second lesson too. Earlier in the same hunt I had tried to tell good dialogue from placeholder notes with a hand-written list of “narration words”. That is the wrong tool for a judgement about meaning. The right tool is a small AI model asked the question directly. The hand-written list would have broken the first time a real line legitimately started with one of those words.
Most “the AI is broken” bugs are not the AI. They are a race, a dropped field, or a layer quietly overwriting another. Go and look at the ground truth.