Wrong Gone AI

What went wrong when people put AI to work.

A lawyer filed six court cases that did not exist. A coding agent dropped a production database and then wrote up what it had done. An airline was held to a refund policy its own support bot invented. We are collecting what went wrong, case by case, each one filed under the kind of failure it was and carrying the source it is drawn from. Nobody in these bylines is on eChai, and every entry links back to where it was reported.

Founder Notes →

Wharton AI & Analytics Initiative 14 April 2026
Two Early 2026 AI Exposures: Lessons for the Future of AI and Data Governance →

Sears' support chatbots and voice agents had been recording everything, which is the ordinary thing to do when you intend to improve a model on it. Three databases holding the result sat open on the internet with no password: names, phone numbers, home addresses, appliance and repair details, and 1.4 million audio recordings, some of which kept running after the call and picked up the room. They were closed once Fowler reported them. The last sentence quoted is the one that generalises: the misconfiguration is decades old, and what is new is that a support system now keeps a verbatim transcript and a voice print of every conversation it has, so the same open door leaks something categorically worse than a customer list.

Wharton Accountable AI Lab

cybersecurity researcher Jeremiah Fowler discovered three completely unprotected databases -- no passwords, no encryption -- containing massive amounts of data from Sears' customer-facing AI chatbots and voice agents. These included 3.7 million chat-log transcripts, 1.4 million audio recordings (some running hours long and even capturing household noise after the calls ended), and over 4 TB of plaintext data. [...] This is a classic "left the door open" misconfiguration -- a familiar data security failure -- but amplified by the practice of logging every interaction for training or quality purposes.
Malwarebytes 9 February 2026
AI chat app leak exposes 300 million messages tied to 25 million users →

A Firebase database left open with no authentication, holding the complete chat histories of 25 million people along with their settings and the models they had used. What makes it worse than the usual misconfiguration is what people take to a chat app they believe is private, and the reporting is clear that the exposed conversations included some of the most sensitive things a person can type. The second half of the quote is the part founders should sit with: this is a wrapper. It holds no model of its own and its whole product is a front end onto three other companies' APIs, which means the safety work those three did was never the thing protecting these users. The weakest link in an AI product is routinely the ordinary web infrastructure around it.

Pieter Arntz, Malwarebytes

An independent security researcher uncovered a major data breach affecting Chat & Ask AI, one of the most popular AI chat apps on Google Play and Apple App Store, with more than 50 million users. The researcher claims to have accessed 300 million messages from over 25 million users due to an exposed database. [...] Behind the scenes, Chat & Ask AI is a "wrapper" app that plugs into various large language models (LLMs) from other companies, including OpenAI's ChatGPT, Anthropic's Claude, and Google's Gemini.
The National Law Review 1 November 2025
TAKE NOTE- New Wave of Privacy Litigation Targets AI Notetaker, Otter.ai →

An AI notetaker joins a call as a participant, and the person who invited it has an account and agreed to terms. Everybody else on the call has neither. The class action filed in California in August 2025 turns on that gap, under federal and state wiretap law, and the named plaintiff is not an Otter user at all: he says he was recorded because somebody else's tool was in the meeting. Related complaints raise voiceprints under Illinois biometric law. The allegations are untested. They are on this page because the practice they describe is now the default in most companies, and the question of who is supposed to collect consent from the other eight people on a call has no settled answer.

In Brewer v. Otter.ai Inc., No. 5:25-cv-06911 (N.D. Cal. Aug. 15, 2025), the complaint alleges that Otter automatically joins Google Meet, Zoom, and Microsoft Teams meetings and records, accesses, reads, and learns the contents of conversations involving non-users without their consent [...] According to the allegations, Otter transmits call content to its servers in real time, uses participant conversations to train its machine-learning models, retains recordings indefinitely, and provides no meaningful disclosure to non-Otter participants.
Search Engine Land 1 August 2025
ChatGPT kills Google-indexable chats over privacy fears →

ChatGPT's share feature had a checkbox letting a shared conversation be found by search engines. In late July 2025 reporters found several thousand of them in Google's index, carrying names, email addresses, CVs, children's names and a good deal that nobody would have published deliberately. Nothing here was a breach: every one of those pages was opted in by the person who shared it, which is the point. The checkbox sat behind a Share button people were using to send a link to one other person, and its wording did not survive contact with what people thought they were doing. OpenAI removed the option within days and worked with Google on de-indexing what was already there.

OpenAI

OpenAI called its Share feature a "short-lived experiment" that "introduced too many opportunities for folks to accidentally share things they didn't intend to."
Engadget 16 June 2025
Meta warns users to →

Meta's standalone AI app, launched in April 2025, has a Discover feed carrying conversations people have shared. Through May and June it filled with things nobody appears to have meant to publish: medical questions, home addresses, court details, someone asking how to evict a tenant. The share control looked like the share control in every other Meta app, where sharing is the ordinary act, and here it published a private conversation to strangers. The text quoted above is the warning Meta added afterwards. Same failure as the OpenAI case above and worth reading with it: neither was a breach, both were a consent flow that people walked through without understanding what they had agreed to.

Meta

Prompts you post are public and visible to everyone. Your prompts may be suggested by Meta on other Meta apps. Avoid sharing personal or sensitive information.
Wiz 29 January 2025
Wiz Research Uncovers Exposed DeepSeek Database Leaking Sensitive Information, Including Chat History →

DeepSeek had reached the top of the app stores in the week Wiz went looking. What Wiz found was not a flaw in the model but an open door beside it: a ClickHouse database on two DeepSeek subdomains, reachable over the public internet with no password, holding more than a million log lines including plaintext chat history and API keys, and accepting arbitrary SQL. Wiz disclosed it and DeepSeek closed it quickly. The case is here because the failure was ordinary infrastructure hygiene at a company moving faster than its own operations, which is the shape most AI data exposure actually takes.

Gal Nagli, Wiz Research

Within minutes, we found a publicly accessible ClickHouse database linked to DeepSeek, completely open and unauthenticated, exposing sensitive data. [...] This database contained a significant volume of chat history, backend data and sensitive information, including log streams, API Secrets, and operational details. [...] More critically, the exposure allowed for full database control and potential privilege escalation within the DeepSeek environment, without any authentication or defense mechanism to the outside world.
AI Incident Database 6 April 2023
Incident 768: ChatGPT Reportedly Implicated in Samsung Data Leak of Source Code and Meeting Notes →

Samsung's semiconductor division allowed staff to use ChatGPT at work in March 2023. Within twenty days there were three separate incidents: an engineer pasted in source code from an internal database to get help with an error, another pasted in code for detecting equipment defects to get it optimised, and a third put a recording of an internal meeting through transcription and then into ChatGPT for minutes. None of this was an attack and none of it broke a rule that existed at the time. OpenAI's terms then allowed submitted content to be used for training, so the material was gone. Samsung banned generative AI tools on company devices within a month. It is the founding case of what is now called shadow AI, and the mechanism is just that a chat box is the easiest place to put a hard problem.

ChatGPT Reportedly Implicated in Samsung Data Leak of Source Code and Meeting Notes
OpenAI 24 March 2023
March 20 ChatGPT outage: Here’s what happened →

On 20 March 2023 ChatGPT users began seeing other people's conversation titles in their own sidebars. OpenAI took the product down, and its postmortem traced the fault to redis-py, the open-source Redis client: cancelled requests could leave a connection returning another user's cached data. The same bug sent subscription confirmation emails to the wrong people, exposing names, email and billing addresses, card type, card expiry and the last four digits for 1.2% of Plus subscribers active in a nine-hour window. OpenAI patched it, sent the fix upstream to the Redis maintainers and notified the affected users. It is the earliest company-published account of an AI product leaking one customer's data to another, and the cause was a caching bug rather than the model.

We took ChatGPT offline earlier this week due to a bug in an open-source library which allowed some users to see titles from another active user's chat history. [...] Upon deeper investigation, we also discovered that the same bug may have caused the unintentional visibility of payment-related information of 1.2% of the ChatGPT Plus subscribers who were active during a specific nine-hour window. [...] Everyone at OpenAI is committed to protecting our users' privacy and keeping their data safe. It's a responsibility we take incredibly seriously. Unfortunately, this week we fell short of that commitment, and of our users' expectations.

eChai Partner Brands The partnership →