Before a Beirut consumer brand signs an AI vendor contract, Dr. Jonah Tebaa runs an audit that most procurement processes skip entirely: he reads the customers' actual messages. Not the vendor's sample set. Not a curated demo transcript. The real inbox, by hand, before anyone from the vendor side is in the room.
The most recent audit he described involved 500 WhatsApp messages pulled from a Beirut consumer brand's support line. His finding was stark: only 195 of them — 39 percent — were written in Arabic close enough to Modern Standard Arabic for a typical Arabic-language AI benchmark to recognize cleanly. The remaining 305 messages, well over half of the brand's real customer traffic, were Lebanese dialect, Arabizi transliteration, or Arabic-English code-switching mid-sentence. Messages like "chou el prix" or "3endkon delivery la Baabda" weren't outliers in his sample. They were the norm.
A Gap Between the Demo and the Inbox
Dr. Tebaa's argument centers on a specific, measurable gap. In this case, the AI vendor's intent-classification demo scored 92 percent accuracy on a curated MSA sample — the kind of showcase most buyers see and sign off on. When the same model was tested against the 305 dialect and Arabizi messages pulled from the client's own inbox, accuracy fell to 54 percent.
He attributes the drop to how most Arabic-language models are built, not to any single vendor's failure. Training corpora for Arabic AI tend to draw heavily from news archives, government publications, and other formal-register sources — material that is written in MSA almost by convention. Arabizi and Levantine dialect largely live outside that corpus, in casual messaging that formal training data doesn't capture. In his framing, a model can score well on Arabic and still be functionally blind to the Arabic a given market's customers actually type.
Redefining the Win, Not Chasing a Bigger Model
Rather than pushing for a larger or custom-trained model, Dr. Tebaa's team narrowed the AI's job. The system was scoped to handle only the traffic it could reliably classify — clean MSA and clearly structured English — while everything flagged as dialect, Arabizi, or ambiguous code-switching was routed to a human queue with a service-level agreement of under two minutes.
The headline number got worse before it got honest: overall containment dropped from the vendor's quoted 92 percent to a production figure of 58 percent. In his view, that's the correct outcome. The original figure described performance on hand-picked text, not on the business's real customers. The lower number reflected what the system could actually do without silently misrouting the majority of the inbox — which, he notes, is what had been generating customer complaints in the first place, long before anyone measured the underlying language mix.
A Checklist Before Any Contract Gets Signed
Out of that engagement, Dr. Tebaa distills a short list of questions he now recommends any operator ask before scoping an Arabic AI deployment:
- What share of real inbound traffic is Arabizi or code-switched, based on an actual sample rather than an assumption?
- Was the vendor's benchmark built from the client's own dialect and phrasing, or from a generic MSA test set?
- What does the system do at low confidence — answer anyway, or escalate to a person?
- Who reviews a sample of escalated messages weekly, and against what SLA?
- Is the contract tied to demo accuracy, or to production containment measured 30 days after launch?
The Real Lesson Is About Scoping, Not Modeling
Dr. Tebaa's broader point is that the project he describes didn't stumble because the underlying model was weak. It stumbled because no one had defined, in writing, which slice of customers' actual language the AI was responsible for handling. He frames the fix not as a technology upgrade but as a measurement discipline: pull a few hundred real messages, tag each one by dialect, Arabizi, MSA, or code-switching, and treat that tally — not the vendor's demo — as the benchmark that matters.