What does The Five-Month Test: What Dr. Jonah Tebaa Says AI Governance Actually Has to Survive mean in practice?
According to Dr. Jonah Tebaa, AI governance must survive the five-month test: reconstructing and defending a past AI-assisted decision entirely from preserved evidence after models, thresholds, and personnel have changed. Drawing on the NIST AI Risk Management Framework and Article 12 of the EU AI Act, Tebaa states systems must automatically capture pinned model versions, frozen input snapshots, timestamped policy thresholds, and system-level blocks preventing auto-decisions in manual-review bands.
Dr. Jonah Tebaa has a standard test he applies to any AI-assisted decisioning system he is asked to review: pick one decision at random from several months back and try to reconstruct it using only what the organization actually kept. Most systems pass the test they were built for. Far fewer pass this one.
In his work advising banks, insurers, and fintechs across the Middle East, Tebaa has watched a familiar pattern play out. A credit-scoring or claims-decisioning tool goes live. It performs well. It explains itself clearly, in the moment, to the underwriter or adjuster reviewing the output. Everyone involved is satisfied that the system is transparent. Then, months later, someone outside the organization, a regulator, a lawyer, an ombudsman, asks about a specific decision from the past, and the transparency that felt so solid turns out to have an expiration date nobody set.
The Question Nobody Asks Until Someone Else Does
Tebaa illustrates the point with a scenario drawn from the kind of case he sees repeatedly in regulated lending. A mid-sized bank's AI-assisted tool scores an SME loan application in early March: 47 out of 100. Under the policy running at the time, that score should have triggered manual review by a human underwriter, not an automatic decline. The system declined it anyway. Nobody notices, because nobody is looking. Five months pass. The model gets retrained twice along the way, for routine reasons. The review-band policy itself gets tightened in an unrelated update. Then a lawyer's letter arrives asking the bank to justify the original decision, with ten business days on the clock.
"The real test of an AI governance program is never how well it explains a decision on the day it is made," Tebaa argues. "The real test is whether it can be reconstructed and defended from evidence alone, months later, after the model, the thresholds, and the people involved have all quietly moved on."
In his account of the reconstruction, the compliance team has to pull six things in sequence: the exact model version that scored the case (not the current one), the input data frozen as it existed that day (not updated later), the score and its stated reasons, the policy threshold as it stood on that date (not today's), any human sign-off on record, and a note reconciling all of it against what has changed since. Pull any of those from the wrong point in time, and the whole reconstruction is unreliable, whichever way it points.
Four Things a Decision Record Has to Survive
What makes Tebaa's framing notable is where he places the failure. It isn't in the model's accuracy, and it isn't in a single underwriter's judgment. It's in the gap between a policy written down and a policy enforced by the system itself. A rule that says scores between 40 and 55 require human review is not a safeguard if the software is still capable of auto-declining a 47. The rule existed on paper. It was not built into the machine.
Tebaa's prescription, drawn from frameworks like the NIST AI Risk Management Framework and the record-keeping provisions in Article 12 of the EU AI Act, comes down to four things a decision record has to contain the moment it is created, not months later when someone asks:
- A model version pinned to the record, so it never has to be inferred after the fact.
- An input snapshot frozen at decision-time, so later updates to a case file cannot quietly rewrite history.
- A policy threshold timestamped to that date, so decisions are judged by the rule that applied then, not the rule that applies now.
- A hard system-level block on any automatic decision that falls inside a band the policy itself reserves for human review.
That last item, in Tebaa's view, is the one boards most consistently underestimate. It is the difference between a policy that exists in a document and a policy the system is physically unable to violate. Built that way, the March gap in his scenario would have surfaced in March, to the team that owns the process, instead of in August, in a lawyer's letter.
His broader point extends past lending. Any institution running AI-assisted decisions in claims, underwriting, or hiring is one delayed challenge away from discovering whether its own governance file would hold up. Tebaa's argument is that the file is either built before the letter arrives or assembled in a scramble after it does, and only one of those is a strategy.
