Make the machine show its work

The most reassuring check in any system is the one that can never fail. It goes green every time, everybody relaxes, and nobody asks where it got its answer key.

That's the problem.

Here's a check you'll find in some form on almost every team. Say a nightly job exports a table to a file and writes a small manifest next to it: rows: 48112. A second job opens the file, counts the rows, compares them with the manifest and posts a green tick to a channel.

Now ask where 48,112 came from. The export job counted the rows it wrote. So if a page of results never came back from the database, those rows are missing from the file and missing from the count. The check compares the export with the export's opinion of itself. It cannot say no.

I call these mirror checks: the expected answer and the actual answer come out of the same place. Once you look, they're everywhere. A snapshot test whose expected output was produced by running the code under test, then approved with one keypress. A health check that reads a status the service writes about itself. A model asked how confident it is.

Nobody writes a mirror check to cheat. We write them because they're cheap and always green, and green is what we want to see. That makes them worse than no check. No check at least leaves you nervous.

Accountants worked this out centuries ago. In 1494 Luca Pacioli published a mathematics textbook in Venice with a section on how merchants keep their books. He didn't invent double-entry bookkeeping. He wrote down what Italian merchants already did: every transaction goes in as debits and credits of equal size, and at the end the two columns must balance.

Even that has a blind spot. A clerk who never records a sale writes neither side, and the books still balance. So accountants also reconcile against the bank statement, a record of the same money kept by somebody else, who has no reason to agree with your clerk.

That's the idea in one sentence. A check is only as good as the independence of its answer key.

Sums that have to hit

The cheapest independent check I know is a sum against a total that someone declared separately. A receipt lists items and prints a total. An order lists line items, and the payment processor reports what it actually charged. A batch header, written by the sender, says twelve documents, and you count what arrived.

The parts must equal the whole. Exactly.

People hate "exactly." Things are messy, they say, give it a tolerance. Fine, pick one that sounds careful: five dollars. Now a missing four-dollar coffee walks through your check forever. A tolerance is a hole cut in the exact shape of the thing you were trying to catch. If the numbers are allowed to differ, find out why and model that reason, instead of blurring the comparison.

The most common honest reason is rounding, and it bites in a specific way. Say your code computes line amounts to three decimals: 10.004, 20.004 and 5.004. The raw sum is 35.012, which rounds to a printed total of 35.01. But the receipt prints each line rounded too, 10.00, 20.00 and 5.00, and those add up to 35.00. Your check summed the raw amounts, rounded once to 35.01, matched the printed total and passed. The customer adds up the printed lines and gets a different number from the one at the bottom.

Compare the numbers as they will be written down. That's what the next person is going to check.

The bathroom scale

The version with a language model is fancier, and far more popular. You ask a model to read a receipt and return the total, with a confidence score next to it. Or you run a second pass that asks the same model, "Are you sure?" (A probability that a classifier computes over a fixed list of answers is a different animal.) A number the model writes about itself is one more field it filled in.

That's a bathroom scale that asks you what you weigh.

The research says the scale lies the way you'd expect. In 2024 Panickssery, Bowman and Feng found that models grading text can recognise their own writing, and grade it up when they do. When they fine-tuned models to get better at recognising themselves, the bias grew in step. A study of AI resume screening by Xu, Li and Jiang gave models a resume they had written and a human's of comparable quality. The larger models picked their own far more often, a self-preference the authors put at 67% to 82%.

Then there's the most famous "are you sure?" ever typed. In 2023 two New York lawyers filed a brief in Mata v. Avianca that cited court opinions ChatGPT had invented. The judge's sanctions opinion records that the one who did the research asked ChatGPT "Is Varghese a real case," and that it told him the case "does indeed exist" and could be found on Westlaw and LexisNexis, the two databases lawyers actually search. Varghese v. China Southern Airlines did not exist. Neither did five other opinions in the brief. The court ordered the two lawyers and their firm to pay a $5,000 penalty, in large part because they kept standing by the fake cases after the court questioned them.

He did check. He checked with the witness.

Make it quote

The fix for a model that reads documents is to make it point. Every field it extracts must carry a verbatim quote from the source: the exact characters on the page that support the answer. Then plain code searches the source for that quote. If the quote is there, the field passes. If it isn't, the field drops to "uncertain" and goes to a person, and nothing the model says about itself can lift it back.

Normalise whitespace and curly quotes, and nothing else, because fuzzy matching is a tolerance with a nicer name. And keep the code dumb on purpose. A string search can't be flattered, argued with or told that the document is "probably" saying something.

For the lawyer, this check is boring: pull up the case yourself. A citation is a quote with an address. Either the address leads to a real opinion with those words in it or it doesn't, and a database lookup settles that faster than any amount of asking.

What a quote can't prove

Be precise about what passed. A quote check proves that the quote exists in the document. It does not prove that the quote is true.

If the receipt is wrong, the quote is faithfully wrong. If the scanner read a 7 as a 1, the model quotes the 1 and the check finds it, because all the check proves is that the model copied the text it was given. The model can also quote a real line for the wrong field, the subtotal as the total, and the string search is perfectly happy. And the document itself is untrusted. A PDF from a stranger is a prompt from a stranger, and anything it says about how it should be read is content, not instructions.

This is why the quote and the sum work as a pair. The quote ties each number to the page. The sum ties the numbers to each other. A subtotal passed off as the total won't balance against the line items. Neither check is clever. They just get their answer keys from different places.

And don't wire the model to retry until the check goes green. Ask enough times and you'll get a quote that exists for an answer that's still wrong. By then the check has stopped measuring the model and started measuring your patience.

Durable is not correct

Most of the reliability money right now goes to a different problem. Temporal sells durable execution: if a worker crashes halfway through a ten-step job, the job resumes where it stopped instead of starting over or doing a step twice. In September it raised $550 million at a $12.55 billion valuation. Its CEO, Samar Abbas, put the pitch well: "Reliability has never been optional, but AI has quickly raised the cost of skipping it."

He's right. But durable means the step finishes. It says nothing about what the step wrote. A step that saves the wrong total finishes too, reliably, every time.

Keeping the machine running and checking its answers are different jobs. Paying for the first doesn't get you the second.

Stop asking the machine how sure it is. Ask for the quote and the sum, and check them with code too dumb to be persuaded.