What does The Reconstruction Test: What Dr. Jonah Tebaa Says AI Decision Systems Must Prove When Disputed mean in practice?
Under Dr. Jonah Tebaa's evidentiary Reconstruction Standard, an organization facing an AI dispute must prove why a decision was reasonable when made, rather than merely logging its final output. Using a composite case where an insurer paid a $340,000 settlement over an $18,000 claim, Dr. Jonah Tebaa argues systems must preserve a hashed snapshot of the model, frozen reference data, verbatim inputs, decision-time rationales, and an exact record of what the human reviewer saw.
One hundred and twenty-six days after a claim was denied, a customer disputes it. The company that made the decision still has the output. It no longer has the decision. That gap, Dr. Jonah Tebaa argues, is where most AI governance programs quietly fail, and almost nobody notices until the day a regulator or a lawyer forces the question.
Dr. Tebaa illustrates the problem with a composite, illustrative case built from a pattern he says recurs across claims, lending, and hiring systems: a regional health insurer running AI-assisted triage on roughly 40,000 claims a month. A human reviewer signs off on each recommendation in an average of 90 seconds. The system performs well by every measure the company tracks. Then one denied claim, tied to a policy exclusion reason code, is disputed 126 days after the fact. A regulator requests the file.
The company can produce the decision itself: denied, reason code cited, timestamp on record. What it cannot produce, in Dr. Tebaa's account, is the conditions under which that decision was actually made. The reference policy database behind the model had been updated three times since. The model had gone through two point-upgrades in the same window. The reviewer's 90-second sign-off had been logged as a single word, approved, with no rationale captured and no record of what that person actually saw on screen at the moment they acted.
Unable to reconstruct the decision as it stood at the time, the company settles rather than defend it. In Dr. Tebaa's composite figures, the settlement runs to $340,000, against an original disputed claim worth roughly $18,000.
An Output Is Not an Explanation
The core distinction Dr. Tebaa draws is between what a system logs and what a dispute actually requires. Nearly every AI system he has reviewed keeps some form of log: a record that the system produced a given output, at a given time, for a given case. That record is accurate, and in his view it is close to worthless once a decision is contested, because it answers a different question than the one being asked.
A dispute does not ask what a system produced. It asks why, under what conditions, at that specific moment, the system produced it, and whether a human who could have caught an error had a genuine opportunity to. Dr. Tebaa puts the distinction plainly: "the system produced this output" is a fact, not a defense. A defense requires being able to show why, given exactly what the model knew and exactly what a reviewer saw, the call was reasonable at the time. Most organizations, in his assessment, have built the first and mistaken it for the second — and the gap stays invisible for as long as no one disputes a decision.
What He Calls the Reconstruction Standard
Dr. Tebaa frames this not as an accuracy standard but as an evidentiary one. A model, in his view, can be exactly right in its output and still leave an organization indefensible, if nobody can show it was right for the reasons now being claimed after the fact. The standard he lays out asks organizations to preserve, at the moment of decision rather than reconstruct afterward:
- The exact model version and configuration in force at that moment, as a dated, hashed snapshot rather than a reference to "the current version."
- The exact reference or knowledge data the model drew on, frozen as it existed then, not as it exists today.
- The full input the model actually received, verbatim, not a paraphrase written after the fact.
- The rationale generated at decision time, stored as produced, never reconstructed from memory or inferred backward from the outcome.
- A time-stamped record of which human reviewed or could have overridden the decision, and exactly what that person saw when they acted.
- A retrieval mechanism that surfaces all of the above together, on demand, without an engineer having to assemble it from separate systems under deadline pressure.
Notably absent from the list, Dr. Tebaa points out, is any line about accuracy. Accuracy, in his framing, is what an organization hopes for going in. Reconstructability is what a dispute actually tests, and it tests it months or years after the fact, long after anyone involved remembers the specifics.
Why the Fix Rarely Gets Built
Dr. Tebaa is direct about an uncomfortable implication of his own case study: a more accurate model would not have solved it. The insurer's triage system may well have made the correct call under the policy in force at the time. The company still paid $340,000, because believing a decision was correct is not, on his account, a defense that holds up without the underlying file.
What makes this notable, in his analysis, is that the standard is not expensive to meet. In a composite build of this kind, storing the frozen model version, the frozen reference data, the verbatim input, and the rationale together runs to roughly $0.004 per decision, with single-digit milliseconds of added latency. At 40,000 claims a month, that works out to on the order of $160 monthly, against a $340,000 settlement the company might otherwise have avoided entirely.
The real barrier, in Dr. Tebaa's telling, is discipline rather than infrastructure. Reference data has to be versioned rather than simply updated in place. Model deployments have to snapshot what changed and when, rather than quietly shipping new weights over old ones. Reviewer sign-off has to capture what a human actually weighed, not just a yes or no. None of it, in his view, is technically difficult. What it requires is a decision, made before any dispute exists, that the decision itself is worth preserving and not just its output — which is precisely why, he argues, almost nobody builds it until the day they desperately need it.
The Standard Before It Ships
Dr. Tebaa's practical recommendation is to apply a single test to any AI-assisted decision system before it goes live: if a specific decision is disputed four months from now, can the organization reconstruct, not just report, exactly what happened for that one case. Not the system's average behavior. That claim, that application, that denial.
If the honest answer is no, he argues, the system is not ready to make decisions with real consequences for real people, regardless of how strong its accuracy numbers look in a demo. In his framing, the reconstruction test belongs at the design stage, not as a checkbox added after a dispute lands — because by the time a case is 126 days old, it is already too late to build the record that would have defended it.
Related evidence: The EU AI Act obliges providers of high-risk AI systems to report a serious incident to the market surveillance authorities immediately after establishing a causal link to the system, and in any event not later than 15 days after becoming aware of it — a disclosure deadline fixed in law rather than decided during the incident. (the EU AI Act's 15-day serious-incident reporting deadline)
The Model Cards paper proposes short documents that accompany trained machine learning models and report benchmarked evaluation across a variety of conditions. (the Model Cards for Model Reporting paper)
