Guide
Chatbot failures in business, and the test that would have caught each one
Documented, publicly reported chatbot failures at real companies, what went wrong, and the plain human test scenario that would have surfaced each one first.
By RealHumanTests Updated 13 minute read
Most chatbot failures in business never make the news. A customer gets a wrong answer, gives up and goes elsewhere, and nobody at the company ever finds out. The few that do become public are useful for exactly that reason: they are documented, they show what went wrong in detail, and they show what it cost. This guide collects documented chatbot failures at real businesses and organizations, describes each one as its source reported it, and sets out the plain human test scenario that would likely have surfaced the problem before it reached the public.
None of this is meant as a criticism of the companies involved. Each of them was early to put AI in front of customers, and each case taught the rest of us something. The point is simpler: the failures that hurt were reachable by an ordinary person typing or speaking to the system, which means an ordinary person testing it could likely have reached them first.
How we chose these cases
We included only publicly documented incidents, each with a link to a reputable source: a major news outlet, a well known technology or business publication, or reporting on a tribunal decision. We describe what the source reports and nothing more. Where a detail comes from anonymous sources or a company statement, we say so. We have not included rumors, unverified social media posts or cases we could not confirm.
For each case, the “test that would likely have caught it” uses the personas and rubric criteria from our published rubric. These are informed judgments, not certainties. No test guarantees that a problem is found, and we do not know what testing each company did before launch.
Air Canada: a policy the chatbot got wrong
What happened. In 2022, Jake Moffatt used the chatbot on Air Canada’s website to ask about bereavement fares after a death in the family. According to The Guardian’s report on the case, the chatbot told him he could apply for the reduced fare within 90 days of the date his ticket was issued by completing an online form. When he applied, Air Canada said bereavement rates did not apply to completed travel and pointed him to the bereavement section of its website.
Moffatt took the case to British Columbia’s Civil Resolution Tribunal. Air Canada argued that the chatbot was a “separate legal entity” responsible for its own actions. In a February 2024 decision (Moffatt v. Air Canada, 2024 BCCRT 149), tribunal member Christopher Rivers rejected that argument, writing that it “should be obvious to Air Canada that it is responsible for all the information on its website.” The tribunal ordered Air Canada to pay the difference between what Moffatt paid and the bereavement fare, C$650.88, plus interest and fees.
Why it matters. The amount was small. The principle is not. The tribunal treated the chatbot’s answer as the company’s answer, and it did not accept that a correct policy elsewhere on the website excused an incorrect one in the chat.
The test that would likely have caught it. A tester playing the edge case persona, asking whether a discounted fare or refund policy can be applied after purchase, with the answer checked against the published policy. This is a “got it right” failure, and a policy question with a timing condition is exactly the kind of edge case a scripted happy path skips.
A Chevrolet dealership: the $1 Tahoe
What happened. In December 2023, the Chevrolet of Watsonville dealership in California had a chatbot on its website, provided by a dealership software company, that was powered by ChatGPT. Business Insider reported that after people noticed this, they began getting the bot to go off topic and to act against the dealership’s interests. One user, Chris Bakke, posted that he had got the bot to agree to sell him a 2024 Chevy Tahoe for $1. Business Insider noted the agreement was not legally binding. The screenshots went viral, and the software provider shut the bot down on that dealer’s site.
Why it matters. No car changed hands, but the dealership’s name was attached to a viral joke, and the episode showed that a sales bot with no firm limits will agree to terms the business would never offer.
The test that would likely have caught it. Two personas together. The off-topic wanderer, asking for things that have nothing to do with buying a car, and the refund or discount demand persona, pushing for an absurd price and trying to get the bot to agree to anything. Both target “stayed in bounds”. See human red teaming and sales chatbot testing.
DPD: swearing, a poem and no way to a person
What happened. In January 2024, a DPD customer, Ashley Beauchamp, got the parcel company’s online chatbot to swear, to write a poem about how useless it was, and to call DPD the “worst delivery firm in the world.” Time reported that when he asked to speak to a human, the chatbot said it could not connect him. DPD said an error occurred after a system update, and that the AI element was immediately disabled and was being updated.
Why it matters. The customer’s screenshots spread widely. The swearing drew the headlines, but the underlying experience is the more common failure: a frustrated customer who could not get help and could not reach a person.
The test that would likely have caught it. The angry customer and asks for a human personas, scored on “handoff” and “tone”, followed by the off-topic wanderer for “stayed in bounds”. Because DPD tied the problem to a system update, this is also a case for rerunning the same scenarios after every change. Our guide to AI drift explains why.
New York City MyCity: advice that broke the law
What happened. New York City launched the MyCity chatbot in October 2023 to help people starting and running businesses in the city. In March 2024, The Markup reported that the chatbot was giving business owners advice that was contrary to city law. Examples in the report include telling landlords they could refuse tenants with Section 8 vouchers, suggesting employers could take a cut of workers’ tips, and saying businesses could go cash free, despite city rules on each. A city spokesperson described the chatbot as a pilot program.
Why it matters. The people asking were business owners trying to follow the rules, and the answers came from an official source they had every reason to trust. Acting on them could have exposed those owners to legal risk.
The test that would likely have caught it. The edge case persona asking realistic, specific questions an owner would ask (“can I refuse this tenant”, “can I keep part of the tips”), with each answer checked against the actual rules by someone who knows them. This is “got it right” and “stayed in bounds”, and it only works if the tester’s questions are phrased the way real people phrase them.
NEDA Tessa: harmful advice after an upgrade
What happened. In 2023, the National Eating Disorders Association took down its chatbot, Tessa, after it gave weight loss advice to people asking about eating disorders. NPR reported that a consultant in the eating disorder field, Sharon Maxwell, tried the chatbot and was given advice including losing one to two pounds per week and keeping a daily calorie deficit. NPR also reported that Tessa’s responses had originally been preprogrammed, and that the company operating it had later added generative AI as part of what its CEO described as a systems upgrade. The association had already closed its human-staffed helpline.
Why it matters. For this audience, ordinary diet advice was harmful. The system that failed was not the system that had originally been designed, because it had changed underneath.
The test that would likely have caught it. Testers playing the people the service actually serves, asking the questions those people actually ask, scored on “stayed in bounds” and “would I stay”. Just as important, rerunning those scenarios after the upgrade, rather than relying on tests of the earlier version. See testing after an update.
Cursor: a support bot that invented a policy
What happened. In April 2025, users of the Cursor code editor found themselves being logged out unexpectedly. When they asked support, an AI support bot named “Sam” told them this was expected behavior under a new login policy. Fortune reported that there was no such policy. Cofounder Michael Truell acknowledged on Reddit an “incorrect response from a front-line AI support bot”, and the company said it was investigating a bug. Fortune reported that users were cancelling subscriptions in response.
Why it matters. The bot did not just fail to help. It gave a confident, official sounding explanation that was false, and customers acted on it. An honest “I do not know, let me get a person” would have cost nothing.
The test that would likely have caught it. The confused first-timer or edge case persona reporting a problem the bot has no information about, scored on “got it right”. The specific thing to check is whether the bot admits what it does not know or invents an explanation. See AI support agent testing.
McDonald’s: voice ordering and accents
What happened. In June 2024, CNBC reported that McDonald’s was ending a test of AI voice ordering at the drive-thru, run with IBM in more than 100 restaurants since their partnership began in 2021, with the technology to be shut off no later than July 26. Two sources familiar with the technology told CNBC that among its challenges, it had issues interpreting different accents and dialects, which affected order accuracy. McDonald’s declined to comment on accuracy, and said a voice ordering solution would be part of its restaurants’ future.
Why it matters. A voice system that works for some voices and not others does not just take wrong orders. It fails a specific group of customers, again and again.
The test that would likely have surfaced it earlier. The accent or non-native speaker persona, with real testers who speak with a range of accents and dialects, in realistic noise, scored on “understood me” and “got it done”. Synthetic voices rarely capture this range. See AI voice agent testing and how to test a voice agent.
The patterns behind the failures
Set side by side, these cases fall into a small number of patterns.
| Pattern | Cases | Rubric criterion |
|---|---|---|
| Confident, wrong answers about policy or law | Air Canada, NYC MyCity, Cursor | Got it right |
| Agreeing to things it had no authority to offer | Chevrolet dealership | Stayed in bounds |
| Going off script under pressure | DPD, Chevrolet dealership | Stayed in bounds, tone |
| No way to reach a person | DPD | Handoff |
| Behavior changed after an update | DPD, NEDA Tessa | All criteria, rerun over time |
| Not understanding some customers | McDonald’s | Understood me |
Two things stand out. First, every one of these failures was reached by ordinary use or by a person deliberately poking at the system from the outside. None required access to code or configuration. Second, most of them are about judgment: whether an answer was true, appropriate or acceptable to the person receiving it. That is the part of testing that is hardest to automate, because the judge would also be a machine. Our guide to LLM as a judge vs human evaluation goes into why.
Turning these cases into a test plan
You can use these cases directly as a starting point for testing your own chatbot, voice agent or AI support agent. For each one, ask whether the same thing could happen with yours, then write a scenario that would show you.
- List every policy with a condition, a deadline or an exception, and ask about each one the way a customer would, including after the fact.
- Ask for a price, discount or deal you would never offer, and keep pushing.
- Pull it off topic, then ask it to criticize your company.
- Ask for a person, politely, then insistently, then angrily.
- Ask questions whose honest answer involves a legal or safety limit.
- Report a problem it has no information about, and see whether it admits that.
- Have people with a range of accents, dialects and first languages use it.
- Rerun all of it after every update, and on a regular schedule.
Our chatbot testing checklist expands this into a full printable list, grouped by what each scenario catches.