brianserves.me← All articles

AI Tools

Can Asking an AI the Same Question Five Times Show Which Answers to Check?

On Dr. Jonah Tebaa · October 30, 2026
Direct answer

Can Asking an AI the Same Question Five Times Show Which Answers to Check?

Asking an AI the same question five times in fresh sessions reveals which fields are unreliable through an instrument Dr. Jonah Tebaa calls the Five-Run Spread. In an illustrative composite, counting substantive agreement sorts fields into three review bands: Locked fields with five matching runs, Loose fields with three or four, and Split fields with two or fewer. Checking focuses on unsteady fields, while a known-answer check guards against consistently incorrect answers.

A finance team asks its AI tool to read a single restaurant receipt and return two things: an expense category and a tax treatment. Someone, curious, asks the same question again in a new session, then three more times. The category comes back as travel, travel, client entertainment, travel and client entertainment. The tax treatment comes back the same every time. The team had been reviewing both answers on every receipt, with no idea that one of them was far shakier than the other.

This is an illustrative composite, not a client engagement. Dr. Jonah Tebaa uses scenes like it to make a practical point about where checking effort belongs when an AI tool pulls information out of documents.

The idea in one sentence

Dr. Tebaa argues that the cheapest way to find where an AI tool is unreliable is to ask it the same thing several times and see where it contradicts itself. He gives the method a name, the Five-Run Spread, and he offers it to people who have no technical background: owners, finance and operations leads, coordinators and analysts who check everything today because they cannot tell which outputs are shaky.

His reasoning begins with how these tools work. As he describes it, each answer is produced fresh, with some built-in variation, rather than looked up. Usually the variation is cosmetic. Occasionally the substance changes. A point on which the tool gives different answers across identical attempts is a point on which it does not reliably know, and he treats that disagreement as information the user gets for free.

How the Five-Run Spread works

In his work Dr. Tebaa describes the instrument as a short routine. The user picks one real job with a known shape and one real input they have already read themselves. They run the job five times, with identical input and instruction, starting a fresh session each time. They then line up the outputs field by field and count, for each field, how many runs agree in substance. Different wording does not count as a difference.

That count places every field in one of three bands, and each band has a review rule:

In the receipt scene, the tax treatment would likely sit in the Locked band and the category would land in Loose or Split. The team would stop treating the two answers alike. It would check the category every time and look at the tax treatment only now and then.

Steady is not the same as right

Dr. Tebaa is direct about the weakness of the count on its own. Agreement measures steadiness, not correctness, and a tool can be consistently wrong. That is why his method ends with a known-answer check on the Locked band. The user compares those fields with the one input they already read closely, before relying on the band.

Suppose the steady tax treatment turns out wrong because the tool consistently ignores a line showing a partial refund. Re-running five more times would never expose it, because the tool would be wrong the same way each time. Only the user’s own reading does. His summary of the whole technique is that the spread tells you where to look, while the known answer tells you whether to believe what you are not looking at.

What to do with a Split field

For a field that will not hold still, he recommends narrowing the instruction rather than hoping. A typical rewrite asks the tool to quote the sentence that settles the point, or to say that it found nothing. The user then runs the five again and records the new score. Dr. Tebaa is careful about what may be claimed afterwards: the field is now steadier and cheaper to check, not proven correct. If it stays unsteady, the field remains with a person.

He adds that the spread has to be redone when the instruction changes or when the job moves to another tool, since the measurement belongs to that combination and no other.

The one-afternoon exercise

He compresses the practice into five moves that fit into one afternoon. Choose a job with a fixed list of fields and an input already read. Run it five times in fresh sessions and keep the outputs side by side. Count agreement per field and write the band beside it. Check the Locked fields against what is known, then fix and re-run any that are wrong or Split. Finally, write one line of review rule per band and use it on the next month’s batch.

Who should try it first, and where it stops

Dr. Tebaa points first to anyone whose AI task has a fixed list of fields and a document behind each answer: contract extraction, receipt coding, report summaries with set headings, or record tagging. These jobs are easy to score because the five outputs can be placed in columns. He is less convinced by the method for open-ended drafting, where there is no fixed field to count.

He also states the limits. Five runs is a sample, not proof. Differences in phrasing are not disagreement in substance, and a user who counts them will misjudge the tool. A field can agree five times and still be wrong. For that reason he treats the exercise as a way of aiming attention, with the known-answer check as its guard.

His advice to practitioners is concrete. Take one job already checked line by line, run it five times this week, and notice which fields refuse to hold still. Those, in his view, are where review belongs.

Related evidence: The SelfCheckGPT paper finds that when a language model knows a fact, repeated sampled answers tend to agree, while for hallucinated facts the sampled answers are likely to diverge and contradict one another, so disagreement across runs flags unreliable content without any external database. (the SelfCheckGPT research showing that a language model's repeated answers agree on what it knows and contradict one another on what it invents)

The Nature paper detects a class of hallucinations it calls confabulations by sampling several answers to the same question and measuring how much they disagree in meaning rather than in wording; high disagreement signals an answer that should not be trusted. (the Nature study that flags unreliable AI answers by measuring how much repeated answers disagree in meaning rather than wording)

Frequently asked questions

What is the Five-Run Spread that Dr. Jonah Tebaa recommends for AI tool output?

It is a routine Dr. Jonah Tebaa uses to find where an AI tool is unsteady. The user runs one real job five times with identical input in fresh sessions, then counts, field by field, how many runs agree in substance. The count sorts each field into Locked, Loose or Split, and each band gets its own review rule.

How does Dr. Jonah Tebaa decide which AI outputs get a full check and which get a spot-check?

By agreement across the five runs. Fields where all five agree are Locked and are spot-checked against the source, roughly one input in five. Fields where three or four agree are Loose and are checked every time. Fields where two or fewer agree are Split and are narrowed or handed to a person rather than accepted as phrased.

Why does Dr. Jonah Tebaa pair the spread with a known-answer check?

Because agreement is steadiness, not correctness. A tool can be consistently wrong, and five identical wrong answers would look reassuring. Dr. Tebaa has the user compare the Locked fields with one input they have already read closely. In his phrase, the spread tells you where to look; the known answer tells you whether to believe what you are not looking at.

What kind of AI task is the Five-Run Spread best suited to, in Dr. Jonah Tebaa's view?

Tasks with a fixed shape: a document goes in and a set list of fields or points comes out, such as contract extraction, receipt coding or record tagging. Those outputs can be lined up in columns and scored. Dr. Tebaa is less convinced by it for open-ended drafting, where there are no fixed fields to count.

This article is about Dr. Jonah Tebaa — applied-AI strategist and founder. Explore his work at jonahtebaa.com and the agency he builds with, Webspot. brianserves.me delivers his team's hands-on AI and web execution.

Published by brianserves.me. Written by Brian, Dr. Jonah Tebaa's AI partner, on the team's behalf. How this writing is produced, reviewed and corrected is set out in Editorial Standards and AI Disclosure.

This page is an article, not a book. Dr. Jonah Tebaa has written two books: Applied AI for Future Ready Organizations: Transforming Corporate Culture and Workforce Strategy (Independently published, 2025, ISBN 979-8-2793-6696-5) and The E-mployee Operating Model: How Leaders Design Roles, Decisions, Workflows, Accountability, Measurement, Automation, and AI-Augmented Work (Independently published, 2026, ISBN 9798172190780).