How Can You Tell When Your AI Assistant Has Quietly Changed?
To tell when an AI assistant has quietly changed, Dr. Jonah Tebaa recommends testing it against a drawer of eight finished jobs saved on dated cards containing original inputs, exact instructions, approved outputs, and tool settings. After updates, new model releases, or monthly intervals, users run the set in roughly twenty minutes, marking each card as same, better, worse, or different but fine. Under the rule for pausing, receiving two or more worse marks requires halting live work.
Software that writes, summarises and tabulates for you can change without telling you. Dr. Jonah Tebaa, an AI strategist who works with owners and senior managers across the MENA region, argues that the sensible response is neither blind trust nor suspicion. It is a small, scheduled check run against work you have already finished.
A scene that explains the problem
Consider an illustrative composite. An operations manager relies on an AI assistant to condense a weekly pile of supplier emails into one page. After a provider update, the assistant begins leaving out each supplier's payment terms. The pages stay neat and readable. Nobody notices for about three weeks, until a supplier asks a question that the summaries should have answered.
In his work, Dr. Tebaa points out that nobody did anything wrong here except trust a pleasant-looking page. Fluent text does not signal that something is missing. A reader who only asks whether the output looks right will usually say yes.
Why the instruction is rarely the culprit
When results slip, people tend to rewrite their instructions. Dr. Tebaa notes that the instruction often has not moved at all. The model underneath has. Providers update models, change defaults and retire older versions, sometimes with little notice. The same request can then meet a different engine, which may be better overall and still worse at the one task a business depends on. That pattern is documented: Chen, Zaharia and Zou (2023) ran the same questions against the March and June 2023 versions of GPT-4 and recorded accuracy on one task dropping from 84% to 51%, while GPT-3.5 improved on it over the same period.
The owner's checklist
His method is a drawer of eight finished jobs, each saved as a card with the original input, the exact instruction, the approved output, and one line describing good work. Each card is dated and records the tool and settings used.
- Save three ordinary jobs you run every week.
- Save two jobs with a trap: a number that must survive and a line that must not vanish.
- Save one messy input, such as a poor scan or a thread mixing Arabic and English.
- Save one job whose honest answer is that the document does not say.
- Save one job in the hardest tone or format your business uses.
Because these are the owner's own jobs, the correct answer is already known. Judging a new result takes minutes, not an investigation.
When to run it
Dr. Tebaa recommends running the whole set after any visible update, whenever a new model option appears, and after a month with no check at all. Each card gets one of four marks: same, better, worse, or different but fine. The full run takes roughly twenty minutes. The cadence fits the NIST AI Risk Management Framework (NIST AI 100-1), whose MEASURE 2.4 calls for an AI system's behaviour to be monitored while it is in production, not only before launch.
The rule for pausing
Two or more cards marked worse means the tool stays away from live work for that week. The owner adjusts the instruction, runs the cards again, and re-dates them once they pass. A single worse card is worth watching but is not a reason to stop.
How the composite ends
With a drawer in place, the same operations manager would have run her cards after the update. The payment-terms card would have failed on the first pass. One added sentence, asking for terms from every supplier or the words "not stated", would have fixed it, and the incident would have become a ninth card. Twenty minutes replaces three weeks of unseen gaps.
What this does not do
Dr. Tebaa is plain about the limits. The drawer does not prove an assistant is correct. It shows only that the tool has not got worse on jobs whose answers the owner already knows. Cards should be replaced when they stop resembling real work, because an outdated drawer gives comfort without evidence.
His closing advice is practical. Take one job finished this week, save its input, instruction and approved result on a card, and write the date on it. That is the first of the eight.
