Dr. Jonah Tebaa's argument about deploying operational AI systems keeps returning to one detail: the difference between a system that performed well on 200 test cases and a system that has proven itself on 200 consecutive live transactions. In his work, that difference is the whole ballgame, and he illustrates it with a composite scenario — not an audited case, as he is careful to note, but a pattern built from repeated observation, with numbers attached so the mechanism is visible rather than abstract.
The Scenario He Uses
The example involves a logistics and trading company that built an AI system to match invoices against purchase orders and receiving reports. The system was tested against roughly 200 historical invoices and performed well, so the company removed its human reviewer on the first day of go-live. Around invoice 340, a vendor's export tool merged two data fields in a way the test set had never contained. The system resolved the resulting ambiguity with high confidence — and resolved it incorrectly, routing a payment against the wrong purchase order. With no reviewer left to catch it, the error sat for three weeks until a routine finance reconciliation surfaced it. The cost of unwinding it, in the illustration, comes to roughly $14,000.
His Underlying Claim
Tebaa's point is not that the model was defective. It is that the company measured readiness against the wrong kind of data. Test sets are curated — someone selects and cleans them to represent the cases a team already anticipated. Production data is not curated by anyone; it arrives as whatever a vendor's export tool, a scanner, or a clerk's habits happen to produce, including merged cells, mixed-language fields, and formatting variations nobody thought to test for because nobody had encountered one yet.
A system, he argues, can be highly accurate on a clean test set and still fail in a way that test set never represented — because the failure lives precisely in the slice of reality the test set did not sample. In his framing, this is a procedural failure rather than a technical one: the team skipped the step between proving a model works in testing and trusting it to run without anyone checking behind it.
This mirrors the EU AI Act's human oversight standard: oversight measures must be commensurate with the risks, level of autonomy and context of use of the system, not fixed to a single calendar date.
The Mechanism: An Agreement Count, Not a Date
What Tebaa proposes in place of a calendar-driven go-live is a parallel run — a period in which the AI processes real transactions alongside the existing human process, with every output compared before the human step is retired. The cutover point, in his framing, is set by a threshold the data has to earn, not a date a project plan assigns in advance. He breaks the mechanism into five decisions that have to be settled before the run starts, not during it:
- What counts as agreement between the AI and the human, defined field by field, with tolerance bands set in advance rather than argued about later.
- A named individual, not a team or a shift, responsible for reviewing every discrepancy within 24 hours.
- A combined volume-and-time threshold that ends the run — for instance, 500 consecutive matches or 15 business days, whichever takes longer — rather than a single calendar date.
- A reset rule: any single high-severity mismatch sends the count back to zero, no matter how close it was to the finish line.
- A named person with explicit authority to delay cutover, deliberately chosen so it is not someone whose incentives are tied to the launch date.
Where It Cuts Against Common Practice
Most organizations, in Tebaa's observation, treat a strong test result as sufficient evidence to go live, and treat the go-live date as fixed once it appears on a project plan. His rule reverses both assumptions. Testing performance answers a narrower question than most teams assume — it speaks to accuracy on curated data, not resilience to whatever production will actually produce. And a go-live date set before the parallel run even begins is, in his view, a decision made without the information it is supposed to rest on.
The reframing is straightforward, though it is not the way most rollout plans are built: a system does not earn the removal of human oversight by demonstrating competence once. It earns it by demonstrating consistency, repeatedly, on the version of reality it will actually be running in.
