Every metric said it worked

The test of how hard the AI should think ended on a claim that looked finished: cheaper seeding hadn't cost quality, and a table backed it.

Then someone read the findings.

The run that every number called a success

A German review pass drew eighteen findings on one chapter. It cost what it should have cost, took the time it should have taken, and correctly identified the chapter's most important defect: the seeding pass had left an English word untranslated in the scene the whole chapter turns on, and three findings were classed critical for exactly that.

It was also unusable.

The proposals read hoechsten Huegel, groesste and Identitaet, with liessen in place of ließen. Thirteen of eighteen proposals had every umlaut and eszett replaced with an ASCII fallback. Accept that batch and you inject broken orthography into the baseline, three critical fixes included.

Nothing in the pipeline saw it: the token counts, the timings and the finding counts the earlier test had used as the quality signal all looked normal. Eighteen findings is eighteen findings whether or not a German reader can use them.

Cost is a number the agent logs. Quality is something a person has to read.

French, run that day on the same prompt, model tier and chapter, was fine.

The review prompt's output contract has a line about JSON safety: don't use typographic quotes, mixed Unicode and ASCII pairs break parsing. It is talking about the quotation marks around one field. The model generalized it to the whole response.

German has a standardized ASCII fallback. ae, oe, ue, ss are a real convention every German speaker recognizes. French has no equivalent. Pushed toward ASCII, German had a fallback and French did not.

The tell is that the same run emitted correct typographic quotation marks inside its own proposals. It was still using Unicode where it had no alternative, and transliterating German because German can be transliterated.

The instructions were the bug

Two rules failed, in opposite directions.

1. One rule contained the defect it existed to prevent. A chapter drew 28 findings where similar chapters drew ten or twelve. Twenty-two were the same fix: a closing quotation mark. German quotation marks are „text“ (low opener and high closer). The seeding pass got the low opener right and closed with an ASCII straight quote instead, 33 times across the book. The chapter (written in my daughter's voice) is almost entirely dialogue, so it has more quotation marks than any other.

The instruction meant to prevent it read:

- Quotation marks: „..." (German style) not "..."

The closing mark in that rule is U+0022, the plain straight quote on a keyboard. The instruction meant to teach the correct pair was written with the wrong character, and the seeding pass was doing exactly what it was told. Twenty-four percent of that book's entire review queue was waste generated by one character in the pipeline's own notes.

Nothing but reading the rule as data could have caught that.

2. The other rule never fired at all. French requires a non-breaking space before a colon, semicolon, exclamation point and question mark, and inside its quotation marks. The French review notes said so, as item thirteen in a numbered watch list at the bottom of the file.

The chapter contained zero non-breaking spaces and twenty-one violations. The review pass flagged none of them.

The same rule moved into a locked block near the top, with the measured miss rate named in the rule itself and one sentence saying a whitespace-only fix is expected and correct here. On the next full-book run:

The model, the defects and the rule's wording stayed the same; only its position and framing changed.

What the numbers looked like afterward

Three editions were rebuilt from scratch against revised notes, and both passes ran again. If the new instructions were only making the reviewer chattier, the totals would go up.

Down across the board, which is the right direction. A better seeding pass leaves fewer defects for review to find.

The more telling check was the quiet chapters. The earlier test established that finding counts track the source text rather than the settings, so chapters that drew almost nothing before should still draw almost nothing.

Success. The drop is concentrated in the chapters where the seeding pass used to leave work behind, which is where a real improvement should show up.

What no prompt could fix

The review pass adds a translator's note when the notes call for one. A note that exists only in the translation makes the translated chapter one block longer than its source, and the review pass pairs source and target by index. One extra block near the middle means every block after it gets reviewed against the wrong source. All three languages independently added a footnote on the same line, all three broke alignment, and nothing caught it until two files were compared by hand.

What the new rules caught

The same pass caught things that would otherwise have shipped.

The seeding pass invented profanity in two of three languages. The source reads fear of "messing" up, with the scare quotes doing the work. Italian rendered it as an obscenity and then added a footnote explaining the obscenity. Spanish did the same with a different word. There is no profanity anywhere in this book.

The review pass caught the Italian one and classed it critical. A human caught the Spanish one and left a note asking whether the word was intended.

Where this leaves the series

This round of work was different in kind. There was no number to add up. The finding counts from the earlier test measured what they measure, and they would have called a run of unusable German a clean success.

The fixes came from reading output one chapter at a time, against sources in languages I mostly cannot read, encoding what was learned back into the instructions, and running again to see whether the instruction changed the behavior. Several times it did not, and only another read showed that.

Cheaper is a measurement, and better is an iteration.

Eighteen commits, a hundred and two new tests, and four editions that now hold together.

The pipeline and its cost stayed the same; the difference is that someone read the output.