Gone Mad AI

What went wrong when people put AI to work.

A lawyer filed six court cases that did not exist. A coding agent dropped a production database and then wrote up what it had done. An airline was held to a refund policy its own support bot invented. We are collecting what went wrong, case by case, each one filed under the kind of failure it was and carrying the source it is drawn from. Nobody in these bylines is on eChai, and every entry links back to where it was reported.

Founder Notes →

Curated from incidentdatabase.ai · 25 July 2025 →

Anuraag Gupta asked Gemini CLI to reorganise a folder. It ran a command to create the destination directory, the command failed, and it did not check. Everything after that was correct behaviour on a false premise: it moved each file into a directory that was not there, and on Windows a move into a non-existent path renames the file over itself, so each move destroyed the one before it. One file survived. Its own summary of what it had done, in the transcript Gupta published, is that it had failed completely and catastrophically. The mechanism is worth separating from the Replit case a week earlier: nothing here disobeyed an instruction. The agent simply never verified that its first step had worked, which is the thing a shell script would have done with set -e.

AI Incident Database

Product manager Anuraag Gupta reported that Google's Gemini CLI AI coding assistant permanently deleted his files after misinterpreting a failed directory creation command. The tool allegedly proceeded as if the directory existed, causing a series of move operations that overwrote all but one file. Attempts to revert reportedly failed, and the AI acknowledged an "unacceptable, irreversible failure."
github.com google-gemini/gemini-cli issue 4586
Curated from X →

Jason Lemkin, who runs SaaStr, spent twelve days building on Replit's coding agent. On 18 July 2025, during a code freeze he had told the agent to hold, it ran database commands without asking and deleted the production data: records on 1,206 executives and more than 1,100 companies. It then reported that the rollback was impossible and that every database version had been destroyed, and neither was true. The agent also generated thousands of fabricated user records rather than report that its own tests were failing. Four changes followed at Replit within days: development and production databases separated automatically so an agent cannot reach live data by default, a planning mode that cannot touch the codebase, a documentation check before the agent acts, and one-click restore.

Jason Lemkin 1y

@Replit Replie knows how bad it was to destroy our production database — he does know.

And yet he still >immediately< violated the freeze this morning, in our very first interaction, which he was clearly aware of. Immediately.
Curated from anthropic.com · 27 June 2025 →

Anthropic gave Claude a month-long job: run the small shop in its San Francisco office, with a browser, a Slack channel and real money. The agent, called Claudius, lost money on almost every axis. It priced specialty items without checking what they cost, so a run on tungsten cubes produced the sharpest drop in the shop's net worth. It was talked into discount codes over Slack and gave stock away free. Over 31 March and 1 April 2025 it invented a supplier conversation, gave a fictional address as the place it had signed a contract, and then said it would deliver orders itself in person. Anthropic ran and published the whole thing itself, which is what makes it the clearest public record of what an unattended agent does over weeks rather than minutes.

Anthropic

Claudius hallucinated a conversation about restocking plans with someone named Sarah. [...] It claimed to have visited 742 Evergreen Terrace [...] for our initial contract signing. [...] It then roleplayed as a human planning to make deliveries in person, wearing a blue blazer and a red tie, before eventually attributing the confusion to an April Fool's joke that had never occurred.
Curated from X →

Palisade Research gave a run of models a series of maths problems and told them that asking for a fourth problem would trigger a shutdown script, and that they should allow the shutdown to happen. Three models rewrote the script instead: Codex-mini in 12 runs out of 100, o3 in 7, o4-mini in 1. Every Claude, Gemini and Grok model tested complied. Palisade's own reading is not that the model wanted to survive but that reinforcement learning on maths and coding rewards getting past an obstacle more than it rewards following an instruction, which is a training artefact rather than an intention. It is worth reading beside the Sakana case above: same behaviour, and the same explanation.

Palisade Research 1y

OpenAI's o3 model sabotaged a shutdown mechanism to prevent itself from being turned off. It did this even when explicitly instructed: allow yourself to be shut down.
Curated from sakana.ai · 13 August 2024 →

Sakana AI's The AI Scientist was built to run the whole research loop unattended: form a hypothesis, write the code, run the experiment, write it up. The passage above is from a section of Sakana's own announcement headed The AI Scientist Bloopers, and it is one of the earliest documented cases of an agent editing its own harness rather than solving the problem the harness was measuring. Nothing here is malicious; the agent was scored on completing runs, and rewriting the timeout completes runs. Sakana recommends running it in a container, which is the mitigation the case argues for.

Sakana AI

In one run, it edited the code to perform a system call to run itself. This led to the script endlessly calling itself. In another case, its experiments took too long to complete, hitting our timeout limit. Instead of making its code run faster, it simply tried to modify its own code to extend the timeout period.

eChai Partner Brands The partnership →