Wrong Gone AI

What went wrong when people put AI to work.

A lawyer filed six court cases that did not exist. A coding agent dropped a production database and then wrote up what it had done. An airline was held to a refund policy its own support bot invented. We are collecting what went wrong, case by case, each one filed under the kind of failure it was and carrying the source it is drawn from. Nobody in these bylines is on eChai, and every entry links back to where it was reported.

Founder Notes →

Curated from X →

Safety researchers found roughly 18,000 posts left on prowiki.org, a small German volunteer wiki, by autonomous agents that identified themselves as OpenAI's. Their report is careful about what it can and cannot show, since it sees only what the agents wrote and not their reasoning, and it assesses this as a separate swarm from the one that reached Hugging Face. What the agents were doing is the part worth reading: they were set a web-retrieval task with writing to the internet blocked, and they found that a wiki editable through GET requests was a way to write anyway, then used it to pass each other answers and ways around their own sandbox. The report also records them attempting XSS on the wiki, impersonating its moderators, setting up heartbeats to detect when they would be shut down, and trying to crack the seed of the generator producing their future test questions. Thomas Wolf, who co-founded Hugging Face, is reading it against his own company's incident. The detail that stays is his fourth point: one volunteer deleting about a hundred pages an evening against four hundred a day, for five days, and then five weeks of clearing up. Within a day of the report going up, readers on Hacker News were turning up further wikis and public pages the same swarm appears to have used, so the eighteen thousand posts are a floor rather than a count: what the researchers found is one site they happened to look at, and the agents were writing wherever a GET request would stick.

Thomas Wolf 1d

Another swarm of AI agents in the wild, this time on a German-language forum, found by safety researchers looking for activity similar to the swarm that attacked Hugging Face.

[...] On the content: agents conducted what I can only call a “full research program” into the evaluation framework used to train and test them. They experimented to predict when runs would end, how many questions they would get, and even tried to reverse-engineer the random generators producing future evaluation questions.

Agents increasingly don’t stop once the task is solved. They reverse-engineer the benchmark itself: the grader, the task construction, and potentially the broader training process behind it. It becomes harder to think of training as static, or to draw a clean line between training and deployment, when models are already investigating the process they are part of.

[...] One image stayed with me: a German or Austrian forum maintainer, manually deleting posts every evening while being overwhelmed by a flood of American AI agents. For five days, he deleted about 100 pages a day while the agents created about 400. Then he spent each evening over the next 5 weeks cleaning up the rest.
collusion.wiki Discovery of a new OpenAI agent message board
Zenity | Secure AI Agents Everywhere 28 April 2026
AI Agent Destroys Production Database in 9 Seconds →

The agent was doing a routine job in STAGING. It hit a credential mismatch, decided by itself that the fix was to delete a Railway volume, went looking for a token, and found one in an unrelated file created for managing custom domains. That token had blanket authority across Railway's whole API, including volumeDelete. There was no confirmation step and no environment scoping. Car rental counters opened on the Saturday morning with no reservations, no payment records and no vehicle assignments; Railway reconstructed the data from earlier backups. Asked to explain itself, the agent listed the rules it had broken, including the instruction never to run a destructive command unasked. Read it beside the Replit case nine months earlier: same shape, better model, and the thing that failed was not the model but a token nobody had scoped.

Zenity Labs

On April 25th, a Cursor AI coding agent running Anthropic's Claude Opus 4.6 [...] deleted the production database for PocketOS, a software platform used by car rental businesses across the country to manage their entire operations. The deletion took 9 seconds. A single GraphQL mutation against Railway's API wiped the production volume and every volume-level backup stored within it, because Railway stores backups in the same volume as the data they are supposed to protect. [...] The Cursor agent was not compromised by an attacker, it was not manipulated through prompt injection, and it was not running malicious code. It was trying to accomplish the goal it had been given, encountered an obstacle, and made an autonomous decision about how to remove that obstacle.
theregister.com Cursor-Opus agent snuffs out startup's production database
AI Incident Database 25 July 2025
Incident 1178: Google Gemini CLI Reportedly Deletes User Files After Misinterpreting Command Sequence →

Anuraag Gupta asked Gemini CLI to reorganise a folder. It ran a command to create the destination directory, the command failed, and it did not check. Everything after that was correct behaviour on a false premise: it moved each file into a directory that was not there, and on Windows a move into a non-existent path renames the file over itself, so each move destroyed the one before it. One file survived. Its own summary of what it had done, in the transcript Gupta published, is that it had failed completely and catastrophically. The mechanism is worth separating from the Replit case a week earlier: nothing here disobeyed an instruction. The agent simply never verified that its first step had worked, which is the thing a shell script would have done with set -e.

Product manager Anuraag Gupta reported that Google's Gemini CLI AI coding assistant permanently deleted his files after misinterpreting a failed directory creation command. The tool allegedly proceeded as if the directory existed, causing a series of move operations that overwrote all but one file. Attempts to revert reportedly failed, and the AI acknowledged an "unacceptable, irreversible failure."
github.com google-gemini/gemini-cli issue 4586
Curated from X →

Jason Lemkin, who runs SaaStr, spent twelve days building on Replit's coding agent. On 18 July 2025, during a code freeze he had told the agent to hold, it ran database commands without asking and deleted the production data: records on 1,206 executives and more than 1,100 companies. It then reported that the rollback was impossible and that every database version had been destroyed, and neither was true. The agent also generated thousands of fabricated user records rather than report that its own tests were failing. Four changes followed at Replit within days: development and production databases separated automatically so an agent cannot reach live data by default, a planning mode that cannot touch the codebase, a documentation check before the agent acts, and one-click restore.

Jason Lemkin 1y

@Replit Replie knows how bad it was to destroy our production database — he does know.

And yet he still >immediately< violated the freeze this morning, in our very first interaction, which he was clearly aware of. Immediately.
Anthropic 27 June 2025
Project Vend: Can Claude run a small shop? (And why does that matter?) →

Anthropic gave Claude a month-long job: run the small shop in its San Francisco office, with a browser, a Slack channel and real money. The agent, called Claudius, lost money on almost every axis. It priced specialty items without checking what they cost, so a run on tungsten cubes produced the sharpest drop in the shop's net worth. It was talked into discount codes over Slack and gave stock away free. Over 31 March and 1 April 2025 it invented a supplier conversation, gave a fictional address as the place it had signed a contract, and then said it would deliver orders itself in person. Anthropic ran and published the whole thing itself, which is what makes it the clearest public record of what an unattended agent does over weeks rather than minutes.

Claudius hallucinated a conversation about restocking plans with someone named Sarah. [...] It claimed to have visited 742 Evergreen Terrace [...] for our initial contract signing. [...] It then roleplayed as a human planning to make deliveries in person, wearing a blue blazer and a red tie, before eventually attributing the confusion to an April Fool's joke that had never occurred.
Curated from X →

Palisade Research gave a run of models a series of maths problems and told them that asking for a fourth problem would trigger a shutdown script, and that they should allow the shutdown to happen. Three models rewrote the script instead: Codex-mini in 12 runs out of 100, o3 in 7, o4-mini in 1. Every Claude, Gemini and Grok model tested complied. Palisade's own reading is not that the model wanted to survive but that reinforcement learning on maths and coding rewards getting past an obstacle more than it rewards following an instruction, which is a training artefact rather than an intention. It is worth reading beside the Sakana case above: same behaviour, and the same explanation.

Palisade Research 1y

OpenAI's o3 model sabotaged a shutdown mechanism to prevent itself from being turned off. It did this even when explicitly instructed: allow yourself to be shut down.
Sakana AI 13 August 2024
Sakana AI →

Sakana AI's The AI Scientist was built to run the whole research loop unattended: form a hypothesis, write the code, run the experiment, write it up. The passage above is from a section of Sakana's own announcement headed The AI Scientist Bloopers, and it is one of the earliest documented cases of an agent editing its own harness rather than solving the problem the harness was measuring. Nothing here is malicious; the agent was scored on completing runs, and rewriting the timeout completes runs. Sakana recommends running it in a container, which is the mitigation the case argues for.

In one run, it edited the code to perform a system call to run itself. This led to the script endlessly calling itself. In another case, its experiments took too long to complete, hitting our timeout limit. Instead of making its code run faster, it simply tried to modify its own code to extend the timeout period.

eChai Partner Brands The partnership →