Gone Mad AI

What went wrong when people put AI to work.

A lawyer filed six court cases that did not exist. A coding agent dropped a production database and then wrote up what it had done. An airline was held to a refund policy its own support bot invented. We are collecting what went wrong, case by case, each one filed under the kind of failure it was and carrying the source it is drawn from. Nobody in these bylines is on eChai, and every entry links back to where it was reported.

Founder Notes →

Curated from techcrunch.com · 30 August 2025 →

Taco Bell put voice ordering into more than 500 drive-throughs from 2023. What followed was a run of clips: orders misheard, orders repeated back wrong, and a customer ordering 18,000 cups of water specifically to break the system and get a human. In August 2025 the chain's chief digital and technology officer told the Wall Street Journal it was reconsidering where the technology belongs, and the answer he described is not removal but supervision: a person on the drive-through at busy sites, voice AI where the queue is short, and franchisees choosing. Quoted here because that is the shape most of these rollbacks actually take, and it rarely gets reported as one.

Dane Matthews, Taco Bell

Sometimes it lets me down, but sometimes it really surprises me. [...] For our teams, we'll help coach them: at your restaurant, at these times, we recommend you use voice AI or recommend that you actually really monitor voice AI and jump in as necessary.
Curated from customerexperiencedive.com · 27 May 2025 →

Duolingo's chief executive published an AI-first memo in April 2025 saying the company would gradually stop using contractors for work AI could handle. The backlash came from customers rather than staff: people said they would leave, the company's own social accounts went quiet for a month, and on the Q2 earnings call von Ahn said the memo had dampened customer growth. He clarified on LinkedIn that the point was a change in how work is approached rather than mass layoffs, and that Duolingo was still hiring. Filed here because the failure is not in the product: the technology worked, the memo was about internal process, and the cost landed entirely on the brand.

Luis von Ahn, Duolingo

when I released my AI memo a few weeks ago, I didn't do that well.
Curated from bloomberg.com · 8 May 2025 →

In February 2024 Klarna said its OpenAI-powered assistant was doing the work of 700 full-time agents, handling 2.3 million conversations in a month and cutting resolution time from eleven minutes to under two. It was the most cited proof point of the year that support could be automated outright, and Klarna's headcount fell by roughly half over the period. In May 2025 its chief executive told Bloomberg the company had gone too far and began recruiting people again, on a remote, flexible model. What Klarna did not do is switch the assistant off: the high-volume tier stays automated and people come back for the complex and the escalated. The number that changed was not deflection rate, it was quality, and it took a year to show.

Sebastian Siemiatkowski, Klarna

We went too far. [...] We focused too much on cost. The result was lower quality. [...] From a brand perspective, from a company perspective, I just think it's so critical that you are clear to your customer that there will always be a human if you want.
Curated from restaurantbusinessonline.com · 17 June 2024 →

McDonald's and IBM began working on automated order taking in 2021 and had it in more than a hundred restaurants. Through 2023 and 2024 clips circulated of it adding hundreds of chicken nuggets to an order, putting bacon on ice cream and refusing to stop. The memo quoted above went to franchisees in June 2024 and the technology came out by the end of July. McDonald's has not abandoned the idea and said it would look for a broader voice ordering solution. Worth reading next to the Taco Bell case, a year later and at five times the scale, which reached the same conclusion by a different route.

Mason Smoot, McDonald's USA

After a thoughtful review, McDonald's has decided to end our current partnership with IBM on AOT and the technology will be shut off in all restaurants currently testing it no later than July 26, 2024.
Curated from blog.google · 23 February 2024 →

Gemini's image generation launched at the start of February 2024 and was withdrawn three weeks later, after it returned racially varied images for prompts where the history is specific, including German soldiers of 1943 and America's founders, and refused a run of ordinary requests. Google paused image generation of people entirely while it reworked the feature. The post is worth reading as an engineering account rather than an apology: a correction applied globally to avoid one failure mode produced a second one, and a safety layer drifted more conservative than anyone had specified. Both are properties of a system tuned after the fact rather than of the model underneath it.

Prabhakar Raghavan, Google

It's clear that this feature missed the mark. Some of the images generated are inaccurate or even offensive. [...] So what went wrong? In short, two things. First, our tuning to ensure that Gemini showed a range of people failed to account for cases that should clearly not show a range. And second, over time, the model became way more cautious than we intended and refused to answer certain prompts entirely -- wrongly interpreting some very anodyne prompts as sensitive. These two things led the model to overcompensate in some cases, and be over-conservative in others, leading to images that were embarrassing and wrong.
Curated from techcrunch.com · 24 October 2023 →

On 2 October 2023 a pedestrian in San Francisco was struck by a car that drove off and thrown into the path of a Cruise robotaxi. The robotaxi stopped, then executed a pull-over manoeuvre, moving about twenty feet with the person underneath it. The footage Cruise showed the California DMV the next day ended at the first stop. The regulator got the rest eleven days later and suspended the company's permits, saying its vehicles were not safe for public operation and that it had misrepresented their safety. Cruise pulled its fleet nationwide, its chief executive resigned, roughly a quarter of staff were laid off, and GM wound the business down in 2024. The vehicle's behaviour was one failure; the eleven days were the one that ended the company.

TechCrunch

The DMV's order of suspension, which TechCrunch has viewed, states that Cruise withheld video footage from an ongoing investigation, prompting the agency to suspend its permit. The DMV said it met with Cruise representatives on October 3, one day after an incident that left a pedestrian, who had initially been hit by a human-driven car, stuck under a Cruise robotaxi.
Curated from technologyreview.com · 18 November 2022 →

Galactica was trained on 48 million papers and offered as a way to summarise literature, write scientific code and draft articles. Within hours people were producing papers with citations that do not exist, wiki entries for compounds that do not exist, and a history of bears in space, all in the register of a journal. Meta pulled the public demo on 17 November 2022. ChatGPT was released two weeks later to a very different reception, which is what makes Galactica worth remembering: the failure mode was identical, and what differed was the claim made for it. Galactica was offered as a way to get scientific facts, and a system that is fluent about everything cannot be that.

Will Douglas Heaven, MIT Technology Review

On November 15 Meta unveiled a new large language model called Galactica, designed to assist scientists. But instead of landing with the big bang Meta hoped for, Galactica has died with a whimper after three days of intense criticism. Yesterday the company took down the public demo that it had encouraged everyone to try out. [...] Meta's misstep -- and its hubris -- show once again that Big Tech has a blind spot about the severe limitations of large language models.
Curated from npr.org · 3 November 2021 →

Zillow Offers bought houses on the strength of Zillow's own price model and resold them. In the third quarter of 2021 it bought 9,680 homes and sold 3,032, and took a 304 million dollar writedown for having paid more than it now expected to get back. Zillow closed the business and cut a quarter of its workforce. This one predates the language model era entirely and is the oldest lesson on the page: the model was not obviously broken, its error was simply more volatile than the business had been built to absorb, and the company had put its own balance sheet behind the prediction. Barton's other line from the same day is the one to remember, that the observed error rate made the company look like a leveraged housing trader rather than the market maker it set out to be.

Rich Barton, Zillow Group

We've determined the unpredictability in forecasting home prices far exceeds what we anticipated and continuing to scale Zillow Offers would result in too much earnings and balance-sheet volatility.

eChai Partner Brands The partnership →