Why Do Customers Come Back After an AI Gave the Correct Answer?
Customers often return after an AI provides a technically correct answer because the response fails to resolve their underlying practical need. Using a diagnostic method called the Return-Visit Autopsy, Dr. Jonah Tebaa analyzes transcript pairs to show why customers follow up within seven days. In an illustrative composite case, return visits stemmed from four distinct failure modes: wrong questions answered, unusable answers lacking actionable steps, unbelieved answers missing authoritative policy sources, and incomplete answers omitting obvious follow-on facts.
Dr. Jonah Tebaa argues that the most honest audit of a customer-facing AI is the group of customers who came back. A business switches on a chat or WhatsApp assistant, the dashboard shows a high resolved rate, and fewer tickets reach the team. In his view, that picture can be accurate and still hide a pattern: some correct answers do not end the matter, and the customers they failed are easy to find.
The Number Behind the Dashboard
Dr. Tebaa opens with a composite, which he states is illustrative and not a real client. A 40-person home-appliance retailer has an assistant that handled 1,200 chats in a month. The dashboard marks 71 percent resolved, or 852 conversations. Yet 163 customers, 13.6 percent of the volume, contacted the business again within seven days.
He treats the choice between those two figures as the real decision. A resolved flag, he notes, usually means the customer pressed a button or stopped writing. Silence may be satisfaction, or it may be a customer who decided a person would do better. Only the returners settle the question, because they have already spent more of their own time to say so.
His central claim is that a customer-facing AI is only as good as the share of its correct answers that end the matter. He proposes a way to find the ones that do not, which he calls the Return-Visit Autopsy.
Reading Pairs, Not Averages
The method starts by listing every customer who contacted the business a second time within seven days of an assistant-handled conversation, matched by phone number, email or account. Each customer yields a pair of transcripts. Dr. Tebaa recommends reading the second one first, since its opening line is often the customer's own summary of what the first conversation lacked.
He does not suggest reading everything. A sample of 40 to 60 pairs, drawn without cherry-picking, is in his view enough, and he describes it as an afternoon and not an analytics project. The sample is small on purpose. The goal is a direction for the next fix, not a figure to publish.
Four Causes
The reader tags the first conversation, the one the assistant handled, with exactly one of four causes. Dr. Tebaa illustrates each with a short fragment from the composite.
- Wrong question answered: the assistant replied to the words and missed the need. A customer asked whether a fridge noise was normal and received the model's decibel rating. Twenty-one of the 60 sampled pairs.
- Right answer, unusable: the answer was correct but gave the customer nothing to act on. The warranty covered the repair, yet no address, form or timeframe followed. Eighteen of 60.
- Right answer, unbelieved: the answer was true but named no policy or source, so the customer checked with a person before spending money. Nine of 60.
- Right answer, incomplete: the answer was true but missed the follow-on fact that was always going to be the next question, such as door clearance for a dishwasher. Twelve of 60.
The four counts add up to the 60 pairs, which he treats as a basic check that nothing was tagged twice or skipped.
What the Count Says to Fix First
Dr. Tebaa reads the split as a prioritisation tool. The largest class, wrong question at 35 percent, is also the slowest to repair, because it means the assistant misreads intent and the conversation design needs work. The cheapest win, in his view, is the unusable class at 30 percent.
In the composite, all 18 unusable cases came from one answer template that confirmed warranty cover and then stopped. Adding two lines, where to send the device and how long a repair takes, addressed the whole class. Applied to the 163 returners, 30 percent is about 49 customers a month, and he estimates the change would remove close to a third of the returns. He is careful to call that an estimate until the review is repeated.
He also points out that the cause mix tells a business who should act. Wrong-question cases go to whoever owns the conversation design. Unusable and incomplete answers go to the person who writes the answer content. Unbelieved answers go to the policy owner. The review, he says, turns into assignments, and the debate about whether the assistant is good gives way to four numbers and three owners.
A Short Checklist and a Fair Comparison
Dr. Tebaa condenses the method into seven steps: define the return window, pull the pairs, remove genuine new topics, tag one cause per first conversation, count, assign by cause owner, and re-pull four weeks later using the same window and method. He emphasises the last step. Without a like-for-like comparison, a business cannot tell whether the fix changed anything.
What the Method Cannot See
He is direct about the limits. Customers who gave up in silence never return, so they never appear in the sample. He suggests pairing the review with a short prompt at the end of a conversation asking whether it solved the problem, so that some of them are heard. In markets where much of the contact happens on WhatsApp, including Lebanon, one customer may write from two numbers, which makes matching slower.
On privacy, Dr. Tebaa asks for care before anyone reads a transcript. Names, phone numbers, addresses and payment details should be redacted, access should be limited to the few people doing the review, and customers should be told in the business's notice that conversations may be reviewed to improve the service. Rules differ by country, and he advises owners to check their own.
One Small Action for This Week
Dr. Tebaa ends with a request that is small and reversible: on a single afternoon, pull 40 pairs, tag them, and write down four numbers. Nothing is changed and nothing breaks if the owner stops. He then invites readers who try it to say which cause dominated, and he expects the answer to differ from what most owners predict.
Related evidence: First contact resolution measures whether a customer's issue is settled without follow-up contacts, and it is best judged from the customer's side, because the customer is the only one who knows whether the issue was resolved. (the first-contact-resolution literature, which holds that only the customer knows whether their issue was actually resolved)
Incentivized experiments find customers avoid a chatbot that may hand them on to a second, expert stage ('gatekeeper aversion'), and recommend transparency about what the bot can handle plus faster live-agent access for customers who experienced chatbot failure. (experimental research on customer-service chatbots, which recommends faster live-agent access for customers who experienced a chatbot failure)