Wrong Gone AI

What went wrong when people put AI to work.

A lawyer filed six court cases that did not exist. A coding agent dropped a production database and then wrote up what it had done. An airline was held to a refund policy its own support bot invented. We are collecting what went wrong, case by case, each one filed under the kind of failure it was and carrying the source it is drawn from. Nobody in these bylines is on eChai, and every entry links back to where it was reported.

Founder Notes →

Rest of World 29 July 2025
Inside the collapse of Builder.ai: Was it even an AI company? →

Builder.ai told customers that an assistant called Natasha would build their app, and told investors the same. It reached a 1.2 billion dollar valuation with Microsoft among its backers. Rest of World spoke to former staff across its offices and found the work was being done largely by human engineers, several hundred of them, mostly in India. The company filed for insolvency in May 2025 after a lender seized 37 million dollars during a revenue investigation; reported 2024 revenue of around 220 million dollars was later put closer to 55 million. Roughly a thousand people lost their jobs. Filed under what it was rather than what it claimed: this is not a model that failed, it is a company that described people as software.

It promised to use AI to radically simplify, and automate, the normally time-consuming and technically tedious process of building products. This was via two signature pieces of tech. The first were pre-coded blocks of reusable features [...] Second was a proprietary AI tool the company advertised as revolutionary, able to drastically reduce human labor and compress "weeks of work into hours and minutes."
U.S. Securities and Exchange Commission 9 April 2025
SEC Charges Founder of AI-Powered Shopping App With Fraud →

Nate's pitch was one tap: the app would complete any e-commerce checkout for you, autonomously. The DOJ indicted its founder and the SEC filed alongside on 9 April 2025, alleging that from 2019 the purchases were being completed by hand by contractors overseas, and that around 42 million dollars was raised on the claim. The company collapsed after a news investigation questioned the technology. Read beside Builder.ai and Presto: three companies, three regulators, one shape. The difference here is the criminal charge, which is what separates a claim that outran the product from one a prosecutor says was never true.

Saniger marketed Nate as a cutting-edge mobile shopping app that worked "like magic" using AI, machine learning and neural networks. [...] the transactions were allegedly being processed manually by contract workers located in the Philippines and elsewhere.
dlapiper.com DOJ and SEC send warning on AI washing with charges against technology startup founder
Federal Trade Commission 18 February 2025
FTC Finalizes Order with DoNotPay That Prohibits Deceptive 'AI Lawyer' Claims, Imposes Monetary Relief, and Requires... →

DoNotPay sold a subscription on the claim that its chatbot substituted for a lawyer. The FTC's complaint, from September 2024, does not say the technology failed; it says nobody ever checked whether it worked. No attorney was retained to test the legal output, and no comparison against a human lawyer was ever run. The order finalised in February 2025 requires 193,000 US dollars, notice to everyone who subscribed between 2021 and 2023, and a bar on claiming any professional substitution without evidence. Read beside the Presto case: the SEC's route is securities fraud and the FTC's is consumer deception, and the failure underneath both is a claim about capability that was never measured.

US Federal Trade Commission

The Federal Trade Commission finalized an order requiring DoNotPay, a company that promoted its online subscription service as "the world's first robot lawyer," to stop making deceptive claims about the abilities of its AI chatbot. [...] the company did not test whether its "AI lawyer" operated to the level of a human lawyer when generating legal documents and giving advice, and the company did not hire or retain attorneys to test the quality and accuracy of its service's law-related features.
U.S. Securities and Exchange Commission 14 January 2025
SEC Charges Restaurant-Technology Company Presto Automation for Misleading Statements About AI Product →

Presto Automation sold voice ordering to restaurant chains and told investors it removed the need for a person to take the order. In its own filings in late 2023 it disclosed that more than 70 percent of orders taken by Presto Voice needed a human agent, working from offshore locations, and that the speech recognition in every deployed unit belonged to somebody else. The SEC charged the company in January 2025 over the misleading statements. This is the case that gave the practice a regulatory name, AI washing, and it is the one to cite when the question is whether over-claiming in a deck is a compliance matter: for a public company, the answer is now yes.

it falsely claimed that its own AI product eliminated the need for human order-taking. In fact, the vast majority of drive-thru orders placed through this version of Presto Voice required human intervention. [...] the AI speech recognition technology in all units of Presto Voice that the company had then deployed was owned and operated by a third party.
Axios 17 April 2024
Amazon says shoppers are not being surveiled in Just Walk Out locations →

Just Walk Out let a shopper take things off a shelf and leave, with the bill arriving later. In April 2024 The Information reported that more than a thousand people, largely in India, were reviewing transactions, and that in 2022 around 700 of every 1,000 sales went through human review. Amazon disputes the framing, and its response is quoted here rather than the report, because on this case the dispute is the substance: nobody argues that people were involved, the argument is about how many and whether they were labelling recordings to train the system or completing the transaction. Amazon removed Just Walk Out from its Fresh grocery stores the same month, replacing it with smart trolleys, while continuing to sell it to third parties. Filed alongside Builder.ai because the shape recurs: what a demonstration shows and what the running system costs are different questions, and only one of them is usually published.

Amazon

Amazon stated that the notion of human reviewers watching live shoppers was "completely not true," and that workers in India watch some videos "after the fact" to teach machines, with "way less than 1,000" people helping in "a small percentage of cases".
Futurism 27 November 2023
Sports Illustrated Published Articles by Fake, AI →

Futurism found product reviews on Sports Illustrated's site published under bylines including Drew Ortiz and Sora Tanaka, with biographies, and with headshots that were for sale on a marketplace for AI-generated portraits. The authors do not exist. When Futurism asked, the bylines came off the site. The publisher, Arena Group, said the material came from a third-party content partner, AdVon Commerce, which had used pen names without authorisation, and ended the partnership. It belongs on this page rather than in a media column because of what the invented byline was doing: the words might have been defensible, and the face and the biography were there to tell a reader a person stood behind them.

Maggie Harrison Dupré, Futurism

There was nothing in Drew Ortiz's author biography at Sports Illustrated to suggest that he was anything other than human. "Drew has spent much of his life outdoors, and is excited to guide you through his never-ending list of the best products to keep you from falling to the perils of nature," it read.

eChai Partner Brands The partnership →