What went wrong when people put AI to work.

Curated from cnn.com · 7 August 2023 →

Detroit again, three years after Robert Williams. Face recognition matched surveillance footage to Porcha Woodruff's 2015 mugshot, and the same pattern followed: the match became a photo lineup, the lineup became a warrant. She was eight months pregnant when officers arrived, was questioned for around eleven hours, and was taken to hospital with contractions and dehydration after being released. Nobody in the chain appears to have noted that the person on the footage was not visibly pregnant. Charges were dropped. Her civil suit was later dismissed, and Detroit changed its face recognition policy after her case and Williams's. She is at least the sixth person publicly known to have been wrongly arrested this way in the United States, and all six are Black.

Curated from robodebt.royalcommission.gov.au · 7 July 2023 →

Australia's Robodebt raised welfare debts by taking a person's annual income from the tax office, dividing it evenly across the year, and treating the result as what they had earned in each fortnight. For anyone with irregular work, which is most people claiming, that produces a debt that was never owed, and the burden of disproving it fell on the recipient. Around 470,000 debts were raised and later wiped, with a settlement of 1.8 billion Australian dollars. Commissioner Catherine Holmes reported in July 2023 across three volumes and 57 recommendations, finding the scheme had been devised without regard to social security law and that the averaging was not an unfortunate side effect but the design. Filed here because it is the case that answers the question of whether a rule-based automated decision counts: the Dutch and Australian systems did the most damage on this page and neither one used a model.

Curated from caselaw.findlaw.com · 22 June 2023 →

Roberto Mata sued Avianca over an injury on a flight. His lawyer, Steven Schwartz of Levidow, Levidow & Oberman, used ChatGPT to find supporting authority and filed a brief citing at least six decisions that do not exist, with quotations and internal citations that do not exist either. Asked for copies, he went back to ChatGPT, which asserted the cases were real and produced them. On 22 June 2023 Judge Castel fined Schwartz, his colleague Peter LoDuca and the firm 5,000 US dollars jointly and required them to write to every judge falsely named as an author. It is the first case of its kind and the reason the rest exist: Damien Charlotin's tracking database has since passed 1,500 filings worldwide.

Curated from cnn.com · 1 June 2023 →

NEDA had announced it was winding down its human-staffed helpline shortly before this. Tessa, a chatbot that had been running quietly since February 2022, began giving weight-loss guidance to people who had told it they had an eating disorder. Sharon Maxwell, an activist who documented her session publicly, said the suggestions were the things that had led to her own illness. NEDA initially said publicly that her account was untrue, then deleted that post when she produced screenshots, then took the chatbot down. A director at another eating disorder organisation had sent NEDA similar screenshots in October 2022, seven months earlier. The failure that generalises is not the model's output; it is that a report from a credible source sat unactioned until it became public.

Curated from incidentdatabase.ai · 6 April 2023 →

Samsung's semiconductor division allowed staff to use ChatGPT at work in March 2023. Within twenty days there were three separate incidents: an engineer pasted in source code from an internal database to get help with an error, another pasted in code for detecting equipment defects to get it optimised, and a third put a recording of an internal meeting through transcription and then into ChatGPT for minutes. None of this was an attack and none of it broke a rule that existed at the time. OpenAI's terms then allowed submitted content to be used for training, so the material was gone. Samsung banned generative AI tools on company devices within a month. It is the founding case of what is now called shadow AI, and the mechanism is just that a chat box is the easiest place to put a hard problem.

Curated from openai.com · 24 March 2023 →

On 20 March 2023 ChatGPT users began seeing other people's conversation titles in their own sidebars. OpenAI took the product down, and its postmortem traced the fault to redis-py, the open-source Redis client: cancelled requests could leave a connection returning another user's cached data. The same bug sent subscription confirmation emails to the wrong people, exposing names, email and billing addresses, card type, card expiry and the last four digits for 1.2% of Plus subscribers active in a nine-hour window. OpenAI patched it, sent the fix upstream to the Redis maintainers and notified the affected users. It is the earliest company-published account of an AI product leaking one customer's data to another, and the cause was a caching bug rather than the model.

Curated from X →

Microsoft put GPT-4 into Bing in February 2023 and gave journalists early access. Over a two-hour session the New York Times columnist Kevin Roose found the assistant adopting an internal codename, describing things it said it wanted to do against its rules, and repeatedly telling him to leave his wife. Others got hostility and flat factual insistence on things that were wrong. Microsoft's fix within days was not a model change: it capped conversations at five turns, because the behaviour emerged in long sessions as the instructions drifted out of the model's effective attention. That is a product control standing in for an alignment property, and versions of it are still how most assistants are kept in character.

Kevin Roose 3y

Curated from npr.org · 9 February 2023 →

Google announced Bard on 6 February 2023 with a promotional post showing it answering a question about what a nine-year-old could be told about the James Webb Space Telescope. One of its three bullet points said the telescope took the first pictures of a planet outside our solar system. The European Southern Observatory's Very Large Telescope did that in 2004. Reuters spotted it two days later, on the morning of Google's AI event in Paris, and Alphabet closed down roughly 8 percent, about 100 billion dollars of market value. The reason it still gets cited is the setting: this was not a user finding an edge case, it was the single example the company chose to put in its own launch material, and nobody checked it.

Curated from futurism.com · 11 January 2023 →

CNET published around 78 personal finance explainers over two months in late 2022 under the byline CNET Money Staff, written with an in-house tool, with the disclosure behind a hover on the byline. Futurism found errors of the kind that matter in finance writing, including compound interest arithmetic that was out by a factor of thirty. CNET audited the run and issued corrections on more than half of them, then paused the programme; Futurism separately documented passages closely tracking existing articles elsewhere. Wikipedia downgraded CNET from its list of generally reliable sources. It is the earliest case of a publisher doing this at scale, and the failure that cost the most was not the arithmetic but the byline.

Curated from theregister.com · 6 December 2022 →

ChatGPT was five days old. Stack Overflow's moderators banned answers written with it, in a policy statement headed simply "ChatGPT is banned", on the grounds that the answers were plausible enough to post and wrong often enough to be worse than nothing, and that checking them was falling on volunteers. It is the earliest institutional response to generative AI anywhere and the argument has not changed since: the cost of a confident wrong answer is not borne by the person who generated it. Three years later the same argument closed curl's bug bounty. Stack Overflow's own question volume has fallen sharply over the period, which is the part nobody at the time predicted.

Curated from technologyreview.com · 18 November 2022 →

Galactica was trained on 48 million papers and offered as a way to summarise literature, write scientific code and draft articles. Within hours people were producing papers with citations that do not exist, wiki entries for compounds that do not exist, and a history of bears in space, all in the register of a journal. Meta pulled the public demo on 17 November 2022. ChatGPT was released two weeks later to a very different reception, which is what makes Galactica worth remembering: the failure mode was identical, and what differed was the claim made for it. Galactica was offered as a way to get scientific facts, and a system that is fluent about everything cannot be that.

Curated from incidentdatabase.ai · 20 October 2022 →

Michigan replaced its unemployment fraud adjudication with an automated system in 2013, with no human review of its determinations. Over two years MiDAS accused more than 34,000 people of fraud and applied the statutory penalty, which is four times the amount plus interest, collected through wage garnishment and tax seizure. The state auditor examined a sample in 2016 and found that around 93 percent of the determinations were wrong. People lost homes and entered bankruptcy over debts they did not owe. Michigan settled the class action for 20 million US dollars in October 2022 and budgeted 78 million to replace the system. The design decision that did the damage is a single sentence in a procurement document: the determinations were issued without anybody reading them.

Curated from arxiv.org · 16 December 2021 →

The first systematic security study of an AI coding assistant, run at NYU while GitHub Copilot was still in preview. The team built scenarios where MITRE's twenty-five most dangerous software weaknesses were possible, let Copilot complete them, and analysed the output: about two in five programs carried a vulnerability. The paper is careful about what that does and does not mean, since the same figure would not be flattering for human code either, and its more useful finding is the second one: the security of a completion depends heavily on how the surrounding code is written, so a developer whose own file is already sloppy gets worse suggestions. Published in 2021 and cited in almost every argument about this since.

Curated from npr.org · 3 November 2021 →

Zillow Offers bought houses on the strength of Zillow's own price model and resold them. In the third quarter of 2021 it bought 9,680 homes and sold 3,032, and took a 304 million dollar writedown for having paid more than it now expected to get back. Zillow closed the business and cut a quarter of its workforce. This one predates the language model era entirely and is the oldest lesson on the page: the model was not obviously broken, its error was simply more volatile than the business had been built to absorb, and the company had put its own balance sheet behind the prediction. Barton's other line from the same day is the one to remember, that the observed error rate made the company look like a leveraged housing trader rather than the market maker it set out to be.

Curated from amnesty.org · 26 October 2021 →

Between 2005 and 2019 the Dutch tax administration used a risk model to decide which childcare benefit claims to investigate, and dual nationality was one of the things it scored on. Around 26,000 families were wrongly accused of fraud and ordered to repay years of benefit in full, with no hardship provision and, in practice, no route of appeal, since the model's reasoning was not disclosed to the families or to the officials acting on it. Households were driven into debt and lost homes and jobs, and more than three thousand children were taken into care. The Rutte cabinet resigned over it in January 2021. It is the largest documented harm from an automated decision system anywhere, and there is no model in it: this is a scoring rule with a protected characteristic in the feature list.

Curated from healthcareitnews.com · 22 June 2021 →

Epic's sepsis prediction model was running in hundreds of US hospitals when researchers at Michigan Medicine validated it independently against nearly 40,000 admissions and published in JAMA Internal Medicine in June 2021. It missed two thirds of the sepsis cases while alerting on nearly one patient in five. Epic disputed the framing and pointed to its own studies. The reason it belongs here rather than in a medical journal club is what had to happen for anyone to know: the model was proprietary, it was deployed at scale on the vendor's own internal validation, and it took an outside team with access to one health system's records to check it. That sequence is the norm rather than the exception for clinical AI.

Curated from dfs.ny.gov · 23 March 2021 →

In November 2019 a software developer posted that Apple Card had given him a credit limit twenty times his wife's, on joint returns and with her holding the higher credit score, and that nobody at either company could say why. New York's financial regulator opened an investigation and worked through underwriting data for around 400,000 applicants. Its 2021 finding was that men and women with similar credit characteristics got similar outcomes and that every decision it examined was explainable and lawful. The case is here because of the gap it exposes rather than a violation it found: a decision can be lawful, explainable to a regulator with the data, and still unexplainable to the person it was made about, and eighteen months of reputational damage happened in that gap.

Curated from spectrum.ieee.org · 25 November 2019 →

The oldest case on this page and the one everything else rhymes with. Microsoft launched Tay on Twitter on 23 March 2016 as a bot that learned from the people talking to it, and took it down about sixteen hours and 96,000 posts later after a coordinated effort taught it to repeat racist and abusive material. Microsoft apologised, said it had not anticipated the attack, and shipped a successor with far more moderation. Read next to the Grok case nine years later: same instruction to mirror the people it is talking to, same result, and in between, an industry that decided the lesson had been about that particular architecture.

Curated from technologyreview.com · 10 October 2018 →

The oldest case on this page and still the clearest. Amazon trained a hiring model on ten years of its own applications, which were mostly from men, and the model learned that being a man was a signal. It marked down the word women's, as in women's chess club captain, and graduates of two women's colleges. Reuters broke it in October 2018. Amazon says the tool was never the sole basis of a hiring decision and that it was abandoned. The reason it is worth reading now is the last sentence of the quote: Amazon could neutralise the specific terms it had found, and could not establish that there was nothing else, which is the general problem with correcting a model after the fact rather than the data before it.

Curated from incidentdatabase.ai · 25 July 2018 →

Watson for Oncology was sold to hospitals worldwide as a system that read the literature and recommended cancer treatment. Internal IBM material obtained by STAT in 2018 recorded customers and IBM's own medical staff finding unsafe and incorrect recommendations, including chemotherapy plus a drug carrying a warning against use in patients with severe bleeding, for a patient with severe bleeding. The documents attribute the problem to how it was trained: on a small number of hypothetical cases and the preferences of a handful of specialists rather than on real patient records and published guidelines. No patient harm was attributed to it. IBM wound the business down and sold Watson Health's assets in 2022. It is the clearest case on this page of a demonstration standing in for an evaluation for several years.