What went wrong when people put AI to work.

Curated from fortune.com · 21 March 2025 →

Arve Hjalmar Holmen is a private individual in Norway with no public profile. He asked ChatGPT who he was, and it returned a detailed account of a serious crime, a conviction and a prison sentence, none of which happened, mixed in with real details about where he lives and the number and sex of his children. The European privacy group noyb filed a complaint with the Norwegian data protection authority in March 2025, arguing that the GDPR's accuracy principle applies to personal data a model produces and that a disclaimer does not satisfy it. OpenAI's position has been that the output came from an earlier version and that current versions search the web, making it less likely. The unresolved question, which every similar complaint runs into, is what a right to rectification means for something that is generated fresh each time rather than stored.

Curated from aclu.org · 28 June 2024 →

A shoplifting at a Detroit watch shop in 2018 left a blurry still from a security camera. Police ran it through face recognition, which returned Robert Williams's expired licence photograph, and then built a photo lineup around that result and showed it to a contractor who had not been in the shop and whose only knowledge of the theft was the same poor footage. This is the first publicly reported wrongful arrest from a face recognition match anywhere. The lawsuit settled on 28 June 2024 with what the ACLU calls the strongest police policy in the United States on the technology: no arrest on a face match without independent evidence, no lineup built from a match alone, mandatory training, and an audit of every case since 2017. The compounding failure is worth naming, because it is a process failure and not a model failure: the match was treated as an identification, and every later step was built on it.

Curated from statnews.com · 15 November 2023 →

nH Predict estimates how long a patient will need in a nursing home by matching them against similar patients. The complaint, filed in Minnesota in November 2023 and named for Gene Lokken, a 91-year-old who had passed away that July, alleges the estimate was used to cut off payment for care that doctors had said was still needed, and puts the tool's error rate at around 90 percent, taking that figure from how often denials were overturned on appeal. UnitedHealth's position is that the tool informs providers and families rather than deciding coverage, and that decisions follow Medicare criteria and plan terms. The case is still running. The detail that carries past this one company is the 1 percent target: an estimate stops being an estimate the moment staff are measured against it.

Curated from content.govdelivery.com · 9 August 2023 →

The first case the EEOC settled over discriminatory hiring software, in August 2023, for 365,000 US dollars split between compensatory damages and back pay. It is worth reading precisely because there is no algorithm to blame: nothing was learned from data and nothing drifted. Somebody wrote a rule with an age in it, the software applied it two hundred times, and the company's position was that this was its recruitment process. The settlement also required an anti-discrimination policy, training, and an injunction against asking applicants for their date of birth. Read next to the Amazon case: one is a model that learned a bias and one is a rule that was typed in, and to the person rejected they are indistinguishable.

Curated from cnn.com · 7 August 2023 →

Detroit again, three years after Robert Williams. Face recognition matched surveillance footage to Porcha Woodruff's 2015 mugshot, and the same pattern followed: the match became a photo lineup, the lineup became a warrant. She was eight months pregnant when officers arrived, was questioned for around eleven hours, and was taken to hospital with contractions and dehydration after being released. Nobody in the chain appears to have noted that the person on the footage was not visibly pregnant. Charges were dropped. Her civil suit was later dismissed, and Detroit changed its face recognition policy after her case and Williams's. She is at least the sixth person publicly known to have been wrongly arrested this way in the United States, and all six are Black.

Curated from robodebt.royalcommission.gov.au · 7 July 2023 →

Australia's Robodebt raised welfare debts by taking a person's annual income from the tax office, dividing it evenly across the year, and treating the result as what they had earned in each fortnight. For anyone with irregular work, which is most people claiming, that produces a debt that was never owed, and the burden of disproving it fell on the recipient. Around 470,000 debts were raised and later wiped, with a settlement of 1.8 billion Australian dollars. Commissioner Catherine Holmes reported in July 2023 across three volumes and 57 recommendations, finding the scheme had been devised without regard to social security law and that the averaging was not an unfortunate side effect but the design. Filed here because it is the case that answers the question of whether a rule-based automated decision counts: the Dutch and Australian systems did the most damage on this page and neither one used a model.

Curated from incidentdatabase.ai · 20 October 2022 →

Michigan replaced its unemployment fraud adjudication with an automated system in 2013, with no human review of its determinations. Over two years MiDAS accused more than 34,000 people of fraud and applied the statutory penalty, which is four times the amount plus interest, collected through wage garnishment and tax seizure. The state auditor examined a sample in 2016 and found that around 93 percent of the determinations were wrong. People lost homes and entered bankruptcy over debts they did not owe. Michigan settled the class action for 20 million US dollars in October 2022 and budgeted 78 million to replace the system. The design decision that did the damage is a single sentence in a procurement document: the determinations were issued without anybody reading them.

Curated from amnesty.org · 26 October 2021 →

Between 2005 and 2019 the Dutch tax administration used a risk model to decide which childcare benefit claims to investigate, and dual nationality was one of the things it scored on. Around 26,000 families were wrongly accused of fraud and ordered to repay years of benefit in full, with no hardship provision and, in practice, no route of appeal, since the model's reasoning was not disclosed to the families or to the officials acting on it. Households were driven into debt and lost homes and jobs, and more than three thousand children were taken into care. The Rutte cabinet resigned over it in January 2021. It is the largest documented harm from an automated decision system anywhere, and there is no model in it: this is a scoring rule with a protected characteristic in the feature list.

Curated from healthcareitnews.com · 22 June 2021 →

Epic's sepsis prediction model was running in hundreds of US hospitals when researchers at Michigan Medicine validated it independently against nearly 40,000 admissions and published in JAMA Internal Medicine in June 2021. It missed two thirds of the sepsis cases while alerting on nearly one patient in five. Epic disputed the framing and pointed to its own studies. The reason it belongs here rather than in a medical journal club is what had to happen for anyone to know: the model was proprietary, it was deployed at scale on the vendor's own internal validation, and it took an outside team with access to one health system's records to check it. That sequence is the norm rather than the exception for clinical AI.

Curated from dfs.ny.gov · 23 March 2021 →

In November 2019 a software developer posted that Apple Card had given him a credit limit twenty times his wife's, on joint returns and with her holding the higher credit score, and that nobody at either company could say why. New York's financial regulator opened an investigation and worked through underwriting data for around 400,000 applicants. Its 2021 finding was that men and women with similar credit characteristics got similar outcomes and that every decision it examined was explainable and lawful. The case is here because of the gap it exposes rather than a violation it found: a decision can be lawful, explainable to a regulator with the data, and still unexplainable to the person it was made about, and eighteen months of reputational damage happened in that gap.

Curated from technologyreview.com · 10 October 2018 →

The oldest case on this page and still the clearest. Amazon trained a hiring model on ten years of its own applications, which were mostly from men, and the model learned that being a man was a signal. It marked down the word women's, as in women's chess club captain, and graduates of two women's colleges. Reuters broke it in October 2018. Amazon says the tool was never the sole basis of a hiring decision and that it was abandoned. The reason it is worth reading now is the last sentence of the quote: Amazon could neutralise the specific terms it had found, and could not establish that there was nothing else, which is the general problem with correcting a model after the fact rather than the data before it.

Curated from incidentdatabase.ai · 25 July 2018 →

Watson for Oncology was sold to hospitals worldwide as a system that read the literature and recommended cancer treatment. Internal IBM material obtained by STAT in 2018 recorded customers and IBM's own medical staff finding unsafe and incorrect recommendations, including chemotherapy plus a drug carrying a warning against use in patients with severe bleeding, for a patient with severe bleeding. The documents attribute the problem to how it was trained: on a small number of hypothetical cases and the preferences of a handful of specialists rather than on real patient records and published guidelines. No patient harm was attributed to it. IBM wound the business down and sold Watson Health's assets in 2022. It is the clearest case on this page of a demonstration standing in for an evaluation for several years.