How Many AI Outputs Must a Manager Read to Catch a Mistake?
To reliably catch a mistake in AI outputs, Dr. Jonah Tebaa recommends sizing a review via his Reading Quota planning shortcut: read approximately three divided by the unacceptable error rate. In Dr. Tebaa's hypothetical, illustrative example of 500 drafted payment reminders carrying an intolerable two percent error rate, reading only 20 letters misses errors roughly 67 percent of the time. Sizing the random review to 150 reads catches an error in about 95 percent of checks.
A weekly review that finds nothing feels like good news. Dr. Jonah Tebaa argues that it often is not news at all. In an invented example, a finance lead at a Beirut building-supplies distributor reads 20 of the 500 payment-reminder letters an AI drafts each week. If 2 in every 100 letters carry a wrong balance, her Friday read looks clean about 67 times in 100. All figures in his example are hypothetical and illustrative, built to show a method.
The false all-clear
Dr. Tebaa starts with multiplication. At a 2 percent error rate, the chance that one letter is clean is 0.98. Twenty clean reads in a row have a chance of 0.98 multiplied by itself twenty times, which comes to 0.6676. A single week therefore looks clean two times in three, even though 10 wrong letters, 0.02 x 500, have gone out.
He then lengthens the run. Four clean weeks in a row happen about 20 percent of the time, one month in five, and the unseen cost in his example is 40 wrong letters at USD 45 each, which is USD 1,800. His point is that the reviewer is not at fault. The review was too small to see the error, and its note, "no issues", records what was read and not what was sent.
From error rate to reading quota
His remedy is to size the read from the risk, not from the time available. He offers a rule of thumb, which he presents as his own planning shortcut and not a cited law: read about 3 divided by the error rate you cannot accept. For a 2 percent line that is 150 reads a week. The chance that 150 reads are all clean, if the rate really were 2 percent, is about 4.8 percent, so an error would be seen roughly 95 times in 100.
Dr. Tebaa is careful about the limit of this reasoning. The probabilities assume every letter is wrong independently of the others. Real errors tend to cluster around a new template, a new product line or a changed price list. He therefore calls his method a planning tool and says it is not research, and that it says nothing about any market or any tool.
Pricing the miss before sizing the read
The first two steps of what he calls the Reading Quota come before any arithmetic. A manager names one checkable error, such as a wrong balance, and writes down what a single miss costs when it reaches a customer. Without that price there is no principled way to say which rate is unacceptable, so the quota would be a guess. In his example the price is USD 45, the weekly cost of the error unseen is USD 450, and the firm decides it will not live with a rate of 2 in 100.
Drawing the reads blind
The reads must come from the whole week, chosen at random. Dr. Tebaa says never to read only the newest outputs, the easiest ones, or those that staff have already flagged, because each of those habits quietly makes the sample look better than the work. He also asks for a change of wording. A clean quota should be recorded as "no error seen at the quota", and never as "no errors", since a review reports what it could see.
Letting a formula do the arithmetic
Reading 150 letters at 2 minutes each takes 300 minutes, or 5 hours. His answer is not to cut the quota but to share or shrink the labour. Three people can read 50 each, 100 minutes apiece. Better still, where the error is calculable, a formula can recompute every balance, which covers the wrong-balance error for all 500 letters, and the team can read a smaller set of 50 for tone and wording. People are then used for judgement, and the formula is used for sums.
In his worked week, the first 150 reads show 3 wrong balances, which is 2.0 percent. New sends are held until the template is fixed, and the following week's 150 reads show none, recorded as "no error seen at the quota". For the full first-person argument, including the table and the Monday action he recommends, read his article on jonahtebaa.com: Twenty Reads a Week Will Miss a Two Percent Error.