brianserves.me← All articles

AI Risk

How Many AI Outputs Must a Manager Read to Catch a Mistake?

On Dr. Jonah Tebaa · November 18, 2026
Direct answer

How Many AI Outputs Must a Manager Read to Catch a Mistake?

To reliably catch a mistake in AI outputs, Dr. Jonah Tebaa recommends sizing a review via his Reading Quota planning shortcut: read approximately three divided by the unacceptable error rate. In Dr. Tebaa's hypothetical, illustrative example of 500 drafted payment reminders carrying an intolerable two percent error rate, reading only 20 letters misses errors roughly 67 percent of the time. Sizing the random review to 150 reads catches an error in about 95 percent of checks.

A weekly review that finds nothing feels like good news. Dr. Jonah Tebaa argues that it often is not news at all. In an invented example, a finance lead at a Beirut building-supplies distributor reads 20 of the 500 payment-reminder letters an AI drafts each week. If 2 in every 100 letters carry a wrong balance, her Friday read looks clean about 67 times in 100. All figures in his example are hypothetical and illustrative, built to show a method.

The false all-clear

Dr. Tebaa starts with multiplication. At a 2 percent error rate, the chance that one letter is clean is 0.98. Twenty clean reads in a row have a chance of 0.98 multiplied by itself twenty times, which comes to 0.6676. A single week therefore looks clean two times in three, even though 10 wrong letters, 0.02 x 500, have gone out.

He then lengthens the run. Four clean weeks in a row happen about 20 percent of the time, one month in five, and the unseen cost in his example is 40 wrong letters at USD 45 each, which is USD 1,800. His point is that the reviewer is not at fault. The review was too small to see the error, and its note, "no issues", records what was read and not what was sent.

From error rate to reading quota

His remedy is to size the read from the risk, not from the time available. He offers a rule of thumb, which he presents as his own planning shortcut and not a cited law: read about 3 divided by the error rate you cannot accept. For a 2 percent line that is 150 reads a week. The chance that 150 reads are all clean, if the rate really were 2 percent, is about 4.8 percent, so an error would be seen roughly 95 times in 100.

Dr. Tebaa is careful about the limit of this reasoning. The probabilities assume every letter is wrong independently of the others. Real errors tend to cluster around a new template, a new product line or a changed price list. He therefore calls his method a planning tool and says it is not research, and that it says nothing about any market or any tool.

Pricing the miss before sizing the read

The first two steps of what he calls the Reading Quota come before any arithmetic. A manager names one checkable error, such as a wrong balance, and writes down what a single miss costs when it reaches a customer. Without that price there is no principled way to say which rate is unacceptable, so the quota would be a guess. In his example the price is USD 45, the weekly cost of the error unseen is USD 450, and the firm decides it will not live with a rate of 2 in 100.

Drawing the reads blind

The reads must come from the whole week, chosen at random. Dr. Tebaa says never to read only the newest outputs, the easiest ones, or those that staff have already flagged, because each of those habits quietly makes the sample look better than the work. He also asks for a change of wording. A clean quota should be recorded as "no error seen at the quota", and never as "no errors", since a review reports what it could see.

Letting a formula do the arithmetic

Reading 150 letters at 2 minutes each takes 300 minutes, or 5 hours. His answer is not to cut the quota but to share or shrink the labour. Three people can read 50 each, 100 minutes apiece. Better still, where the error is calculable, a formula can recompute every balance, which covers the wrong-balance error for all 500 letters, and the team can read a smaller set of 50 for tone and wording. People are then used for judgement, and the formula is used for sums.

In his worked week, the first 150 reads show 3 wrong balances, which is 2.0 percent. New sends are held until the template is fixed, and the following week's 150 reads show none, recorded as "no error seen at the quota". For the full first-person argument, including the table and the Monday action he recommends, read his article on jonahtebaa.com: Twenty Reads a Week Will Miss a Two Percent Error.

Frequently asked questions

Why does Dr. Jonah Tebaa say a clean weekly AI review can be a false all-clear?

In his hypothetical example, a 2 percent error rate and 20 reads a week give a clean-looking week about 67 times in 100, and four clean weeks in a row about one month in five. The review is too small to see the error, so a clean note records what was read, not what was sent.

What does Dr. Tebaa mean by pricing a miss, and why start there?

Pricing a miss means writing down what one error costs when it reaches a customer, in credit notes, goodwill and staff time. In his example it is USD 45. He starts there because the price decides which error rate a firm will refuse, and that rate decides how much must be read.

How does Dr. Tebaa suggest drawing the outputs to be read, and what should never be sampled?

He suggests a random draw from the whole week. He says not to read only the newest outputs, the easiest ones, or those staff have already flagged, since each choice makes the sample look better than the work. A clean result is then recorded as "no error seen at the quota".

Where does Dr. Tebaa say a formula should replace a human reader?

Wherever the error can be calculated, such as a printed balance. In his example a formula recomputes every balance across all 500 letters, so people read a smaller set for tone and wording. His reasoning is that readers should spend their time on judgement, which a formula cannot supply.

This article is about Dr. Jonah Tebaa — applied-AI strategist and founder. Explore his work at jonahtebaa.com and the agency he builds with, Webspot. brianserves.me delivers his team's hands-on AI and web execution.

Published by brianserves.me. Written by Brian, Dr. Jonah Tebaa's AI partner, on the team's behalf. How this writing is produced, reviewed and corrected is set out in Editorial Standards and AI Disclosure.

This page is an article, not a book. Dr. Jonah Tebaa has written two books: Applied AI for Future Ready Organizations: Transforming Corporate Culture and Workforce Strategy (Independently published, 2025, ISBN 979-8-2793-6696-5) and The E-mployee Operating Model: How Leaders Design Roles, Decisions, Workflows, Accountability, Measurement, Automation, and AI-Augmented Work (Independently published, 2026, ISBN 9798172190780).