When Is It Fair to Read the Result of an AI Project, According to Dr. Jonah Tebaa?
According to Dr. Jonah Tebaa, evaluating an AI project is fair only after reaching a pre-committed read trigger under his Read Date Rule, rather than during the initial learning dip. Under this method, an official reading is allowed only when three consecutive weeks stabilize within ten percent of each other, occurring never before week six and no later than week twelve. In an illustrative composite case, reporting the average of that steady window prevents premature cancellation.
An owner approves an AI tool, and a few weeks later someone asks whether it is working. Dr. Jonah Tebaa, an AI strategist and author who advises leadership teams across Lebanon and the wider MENA region, argues that the honest answer often depends less on the tool than on the date the question is asked. He sets out his reasoning in his original post, and this page walks through it.
The Message at the End of Week 3
Dr. Tebaa opens with a trading company of 40 people in Beirut. At the end of the third week of an AI rollout, the owner receives a one-line message from his operations manager: the AI made quote packages slower, so should the company stop? Before launch, preparing a supplier quote package took 45 minutes. In week 3 it took 58, which is 29 percent worse.
He is explicit that the company, the owner and every number in the story are an illustrative composite built to demonstrate a method. They are not a real client or a real case.
One Project, Three Opposite Verdicts
In the composite, the early weeks run slow because staff check every line while they learn the tool. The first two weeks average 61 minutes, week 3 is 58, and the figures then fall through 49, 41 and 35. Weeks 6 to 8 read 35, 33 and 34. The average of that steady window is exactly 34 minutes. Dr. Tebaa shows the arithmetic: (45 minus 34) divided by 45 gives roughly 24 percent better than baseline.
The point he draws is that the same project, team and metric support three different conclusions depending on the week.
- In week 3, the reading says stop.
- In week 6, a single good week says maybe.
- Across weeks 6 to 8, the steady average says scale.
He describes the reading error as running in both directions. A pessimistic reading taken during the learning dip can end a tool that was about to pay off. An optimistic reading of the best week can approve a mediocre one. In the composite, quoting only the best week of 33 minutes would claim 27 percent improvement, overstating the real figure by about two points.
The Read Date Rule
Dr. Tebaa's remedy is to fix the date, and the rule that chooses it, in writing before launch. He calls the method the Read Date Rule and gives it five steps.
- Name the one unit. A single all-in measure, such as minutes per completed job, with checking and rework counted.
- Predict the shape. Write down the expected depth of the dip, its length in weeks, and the expected steady value.
- Set the read trigger. The first official reading is allowed only when three consecutive weeks fall within 10 percent of each other, and never before week 6. Week 12 is a hard stop.
- Read the window, not the best week. Report the average of the steady weeks.
- Pre-commit the verdict. Set thresholds for scale, fix and stop before any result exists.
In his composite, the thresholds are scale at 20 percent better or more, fix between 5 and 20 percent, and stop below 5. A 24 percent result therefore means scale, and the decision does not need to be argued afterwards.
Why a Forecast Changes the Week-3 Message
One of the quieter arguments in the piece concerns the forecast. If the owner had written before launch that the early weeks could run as much as 40 percent worse for about four weeks, then a 29 percent reading in week 3 sits inside the expected dip. In Dr. Tebaa's framing, the message stops being an alarm and becomes a checkpoint on a curve the owner drew himself.
When the Curve Does Not Flatten
He treats a curve that is still swinging at week 12 as a finding in its own right. It suggests the tool is not yet stable in the company's process. His advice is to choose between fix and stop using the thresholds already written down, and to extend only with a stated reason, a named change and one new date. Extending by default, he says, simply moves the verdict to a day chosen after seeing more numbers.
The practical instruction he ends on is short: write the read date on the launch document this week.
Related evidence: Evan Miller's widely cited guide to experiment design warns that a result read whenever it first looks good is unreliable, because the significance calculation assumes the sample size was fixed in advance; his remedy is to decide the stopping point before the test starts and not believe the numbers until it is reached. (Evan Miller's guide on why the stopping point must be fixed in advance)
Brynjolfsson, Rock and Syverson's NBER paper on the Productivity J-Curve shows that a new general-purpose technology like AI needs complementary investment in new processes and skills, which leads to an underestimation of productivity growth in the early years of the technology and an overestimation later, once those investments are harvested. (the NBER Productivity J-Curve paper on why early readings of a new technology understate it)
