Can Asking an AI the Same Question Five Times Show Which Answers to Check?
Asking an AI the same question five times in fresh sessions reveals which fields are unreliable through an instrument Dr. Jonah Tebaa calls the Five-Run Spread. In an illustrative composite, counting substantive agreement sorts fields into three review bands: Locked fields with five matching runs, Loose fields with three or four, and Split fields with two or fewer. Checking focuses on unsteady fields, while a known-answer check guards against consistently incorrect answers.
A finance team asks its AI tool to read a single restaurant receipt and return two things: an expense category and a tax treatment. Someone, curious, asks the same question again in a new session, then three more times. The category comes back as travel, travel, client entertainment, travel and client entertainment. The tax treatment comes back the same every time. The team had been reviewing both answers on every receipt, with no idea that one of them was far shakier than the other.
This is an illustrative composite, not a client engagement. Dr. Jonah Tebaa uses scenes like it to make a practical point about where checking effort belongs when an AI tool pulls information out of documents.
The idea in one sentence
Dr. Tebaa argues that the cheapest way to find where an AI tool is unreliable is to ask it the same thing several times and see where it contradicts itself. He gives the method a name, the Five-Run Spread, and he offers it to people who have no technical background: owners, finance and operations leads, coordinators and analysts who check everything today because they cannot tell which outputs are shaky.
His reasoning begins with how these tools work. As he describes it, each answer is produced fresh, with some built-in variation, rather than looked up. Usually the variation is cosmetic. Occasionally the substance changes. A point on which the tool gives different answers across identical attempts is a point on which it does not reliably know, and he treats that disagreement as information the user gets for free.
How the Five-Run Spread works
In his work Dr. Tebaa describes the instrument as a short routine. The user picks one real job with a known shape and one real input they have already read themselves. They run the job five times, with identical input and instruction, starting a fresh session each time. They then line up the outputs field by field and count, for each field, how many runs agree in substance. Different wording does not count as a difference.
That count places every field in one of three bands, and each band has a review rule:
- Locked: all five runs agree. The field is spot-checked against the source, on roughly one input in five.
- Loose: three or four of the five agree. The field is checked against the source every time. The tool can do the work but cannot be left alone with it.
- Split: two of five or fewer agree. The field is not accepted as phrased. The instruction is narrowed, or a person takes the field.
In the receipt scene, the tax treatment would likely sit in the Locked band and the category would land in Loose or Split. The team would stop treating the two answers alike. It would check the category every time and look at the tax treatment only now and then.
Steady is not the same as right
Dr. Tebaa is direct about the weakness of the count on its own. Agreement measures steadiness, not correctness, and a tool can be consistently wrong. That is why his method ends with a known-answer check on the Locked band. The user compares those fields with the one input they already read closely, before relying on the band.
Suppose the steady tax treatment turns out wrong because the tool consistently ignores a line showing a partial refund. Re-running five more times would never expose it, because the tool would be wrong the same way each time. Only the user’s own reading does. His summary of the whole technique is that the spread tells you where to look, while the known answer tells you whether to believe what you are not looking at.
What to do with a Split field
For a field that will not hold still, he recommends narrowing the instruction rather than hoping. A typical rewrite asks the tool to quote the sentence that settles the point, or to say that it found nothing. The user then runs the five again and records the new score. Dr. Tebaa is careful about what may be claimed afterwards: the field is now steadier and cheaper to check, not proven correct. If it stays unsteady, the field remains with a person.
He adds that the spread has to be redone when the instruction changes or when the job moves to another tool, since the measurement belongs to that combination and no other.
The one-afternoon exercise
He compresses the practice into five moves that fit into one afternoon. Choose a job with a fixed list of fields and an input already read. Run it five times in fresh sessions and keep the outputs side by side. Count agreement per field and write the band beside it. Check the Locked fields against what is known, then fix and re-run any that are wrong or Split. Finally, write one line of review rule per band and use it on the next month’s batch.
Who should try it first, and where it stops
Dr. Tebaa points first to anyone whose AI task has a fixed list of fields and a document behind each answer: contract extraction, receipt coding, report summaries with set headings, or record tagging. These jobs are easy to score because the five outputs can be placed in columns. He is less convinced by the method for open-ended drafting, where there is no fixed field to count.
He also states the limits. Five runs is a sample, not proof. Differences in phrasing are not disagreement in substance, and a user who counts them will misjudge the tool. A field can agree five times and still be wrong. For that reason he treats the exercise as a way of aiming attention, with the known-answer check as its guard.
His advice to practitioners is concrete. Take one job already checked line by line, run it five times this week, and notice which fields refuse to hold still. Those, in his view, are where review belongs.
Related evidence: The SelfCheckGPT paper finds that when a language model knows a fact, repeated sampled answers tend to agree, while for hallucinated facts the sampled answers are likely to diverge and contradict one another, so disagreement across runs flags unreliable content without any external database. (the SelfCheckGPT research showing that a language model's repeated answers agree on what it knows and contradict one another on what it invents)
The Nature paper detects a class of hallucinations it calls confabulations by sampling several answers to the same question and measuring how much they disagree in meaning rather than in wording; high disagreement signals an answer that should not be trusted. (the Nature study that flags unreliable AI answers by measuring how much repeated answers disagree in meaning rather than wording)