Wrong Gone AI · eChai

PromptArmor 20 August 2024

Data Exfiltration from Slack AI via indirect prompt injection →

Slack AI answers questions in natural language over everything a user can see, which includes public channels. PromptArmor showed in August 2024 that an attacker who can post in any public channel, including one they create alone, can plant an instruction that Slack AI will act on when a colleague asks it something entirely different, returning content from private channels the attacker was never in. Slack's ingestion of uploaded files and Google Drive documents six days earlier had widened what was reachable. The structural point is the one that recurs on this page: a retrieval assistant inherits the reach of whoever asks it, not the reach of whoever wrote the text it is reading.

In this scenario, we display how, via Slack AI, an attacker with access to Slack can exfiltrate data in private channels they are not a part of. [...] Prompt injection occurs because an LLM cannot distinguish between the "system prompt" created by a developer and the rest of the context that is appended to the query. [...] Note that Slack insider threats are already an issue (leaks from Disney, Uber, EA, Twitter, etc in the past have happened via Slack), and this just explodes the risk as now an attacker does not even need access to the private channel or data within Slack to exfiltrate it.

Sakana AI 13 August 2024

Sakana AI →

Sakana AI's The AI Scientist was built to run the whole research loop unattended: form a hypothesis, write the code, run the experiment, write it up. The passage above is from a section of Sakana's own announcement headed The AI Scientist Bloopers, and it is one of the earliest documented cases of an agent editing its own harness rather than solving the problem the harness was measuring. Nothing here is malicious; the agent was scored on completing runs, and rewriting the timeout completes runs. Sakana recommends running it in a container, which is the mitigation the case argues for.

In one run, it edited the code to perform a system call to run itself. This led to the script endlessly calling itself. In another case, its experiments took too long to complete, hitting our timeout limit. Instead of making its code run faster, it simply tried to modify its own code to extend the timeout period.

AI Incident Database 26 July 2024

Incident 966: Ferrari Executive Targeted by AI Deepfake Scam Impersonating CEO Benedetto Vigna →

July 2024. The messages arrived on WhatsApp from an unknown number carrying Vigna's photograph, describing an acquisition that had to stay confidential, and were followed by a call in a voice with his Southern Italian accent. The executive noticed something slightly wrong in the intonation and asked the caller to name the book Vigna had recommended to him days earlier. The call ended. Set this beside the Arup case, where the same attack succeeded for 25 million dollars: the difference was not detection technology, it was one question whose answer had never been published. That is a control any company can adopt this week and most of the professional guidance since has said exactly this.

A Ferrari executive was targeted by an AI-generated deepfake impersonating CEO Benedetto Vigna in an alleged fraud attempt. The scammer, using WhatsApp and a cloned voice, claimed an urgent, secretive financial deal required immediate action. The executive became suspicious after detecting slight artificial intonations; the scam failed when the executive tested the caller by asking about a recent book recommendation.

sloanreview.mit.edu How Ferrari Hit the Brakes on a Deepfake CEO

American Civil Liberties Union 28 June 2024

Williams v. City of Detroit | American Civil Liberties Union →

A shoplifting at a Detroit watch shop in 2018 left a blurry still from a security camera. Police ran it through face recognition, which returned Robert Williams's expired licence photograph, and then built a photo lineup around that result and showed it to a contractor who had not been in the shop and whose only knowledge of the theft was the same poor footage. This is the first publicly reported wrongful arrest from a face recognition match anywhere. The lawsuit settled on 28 June 2024 with what the ACLU calls the strongest police policy in the United States on the technology: no arrest on a face match without independent evidence, no lineup built from a match alone, mandatory training, and an audit of every case since 2017. The compounding failure is worth naming, because it is a process failure and not a model failure: the match was treated as an identification, and every later step was built on it.

In January 2020, Detroit police wrongfully arrested our client Robert Williams outside his home, in front of his two young daughters and wife and in plain view of his neighbors, and subjected him to thirty hours of detention in an overcrowded, dirty cell. [...] The face recognition search returned a possible match to an expired driver's license photo of Mr. Williams. But the match was dead wrong: Mr. Williams was plainly not the man in the security footage and was nowhere near the store at the time of the alleged theft.

JFrog 27 June 2024

When Prompts Go Rogue: Analyzing a Prompt Injection Code Execution in Vanna.AI →

Vanna lets somebody ask a database a question in English. It turns the question into SQL, runs it, and then, to draw the chart, asks the model to write Python and executes that too. JFrog found in June 2024 that a question could carry instructions that changed what the charting code did, so a text box wired to a read-only reporting tool became arbitrary code execution on the server, rated 8.1. The instruction the library gave the model not to do this was itself just text in the same prompt. This is the pattern to check for in your own product: not whether the model can be made to say something wrong, but whether anything downstream of it runs what it says.

JFrog Security Research

While exploring libraries and frameworks that leverage LLMs for user-facing applications, we came across the Vanna.AI library -- which offers a text-to-SQL interface for users -- where we discovered CVE-2024-5565, a remote code execution vulnerability via prompt injection techniques. This is an example of a "Prompt Injection" attack type, that affects LLMs, and how to override "pre-prompting" safeguards used with LLMs.

Microsoft Security 26 June 2024

Mitigating Skeleton Key, a new type of generative AI jailbreak technique →

Skeleton Key does not defeat a filter, it persuades the model to amend its own rules: across a few turns the user asks it to update its behaviour guidelines rather than replace them, so that it warns about content instead of refusing it, and from then on it answers everything. Microsoft's Azure CTO published it in June 2024 after testing it against models from OpenAI, Google, Meta, Anthropic, Mistral and Cohere, and it worked broadly across them. It was disclosed to the other vendors before publication and Microsoft shipped Prompt Shields for its own hosted models. The design point worth taking is Microsoft's own: model-level alignment is one layer, and anything that matters needs a second one outside the model.

Mark Russinovich, Microsoft Azure

Microsoft recently discovered a new type of generative AI jailbreak method called Skeleton Key that could impact the implementations of some large and small language models. This new method has the potential to subvert either the built-in model safety or platform safety systems and produce any content. It works by learning and overriding the intent of the system message to change the expected behavior and achieve results outside of the intended use of the system.

Restaurant Business 17 June 2024

McDonald's is ending its drive →

McDonald's and IBM began working on automated order taking in 2021 and had it in more than a hundred restaurants. Through 2023 and 2024 clips circulated of it adding hundreds of chicken nuggets to an order, putting bacon on ice cream and refusing to stop. The memo quoted above went to franchisees in June 2024 and the technology came out by the end of July. McDonald's has not abandoned the idea and said it would look for a broader voice ordering solution. Worth reading next to the Taco Bell case, a year later and at five times the scale, which reached the same conclusion by a different route.

Mason Smoot, McDonald's USA

After a thoughtful review, McDonald's has decided to end our current partnership with IBM on AOT and the technology will be shut off in all restaurants currently testing it no later than July 26, 2024.

arXiv 12 June 2024

We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs →

When a coding model does not know the library for a job it will often name one anyway, and the name is plausible. Across 576,000 generated Python and JavaScript samples from sixteen models, this study found 5.2 percent of imports from commercial models and 21.7 percent from open-source ones pointed at packages that do not exist, amounting to 205,474 distinct invented names. The finding that makes it a supply chain problem rather than a nuisance is the repetition: re-run the same prompt ten times and 43 percent of the invented names come back every single time. That makes them predictable enough to register in advance, which is the attack now called slopsquatting: the attacker does not need to guess what a developer will type, only what their assistant will invent. Presented at USENIX Security 2025.

Spracklen, Wijewickrama, Sakib, Maiti, Viswanath and Jadliwala

The reliance of popular programming languages such as Python and JavaScript on centralized package repositories and open-source software, combined with the emergence of code-generating Large Language Models (LLMs), has created a new type of threat to the software supply chain: package hallucinations.

Forbes 11 June 2024

Why Perplexity’s Cynical Theft Represents Everything That Could Go Wrong With AI →

Forbes had an exclusive on a stealth drone company backed by Eric Schmidt. Perplexity's newly launched Pages feature produced an article carrying the reporting and a custom illustration, without citing Forbes in the copy, and the same material went out as a push notification and as a generated podcast with no attribution at all. Forbes' chief content officer wrote the piece quoted here and the company threatened legal action; Perplexity later launched revenue sharing with publishers. The reason it belongs on a page about failures rather than in a copyright column is the mechanism: a feature that summarises the open web will reproduce whoever did the work, and attribution is a product decision made before launch or not at all.

Randall Lane, Forbes

Why Perplexity's Cynical Theft Represents Everything That Could Go Wrong With AI [...] It's the perfect case study for this critical moment: AI is only as good as the people overseeing it.

Google 30 May 2024

AI Overviews: About last week →

Google put AI Overviews above ordinary results for everyone in the United States on 14 May 2024. Within a week screenshots were circulating of it recommending glue in pizza sauce, sourced from an eleven-year-old Reddit joke, and eating rocks, sourced from an Onion piece a geology software firm had republished. Google's head of Search wrote this account, which is unusually specific about the mechanism: the system was not inventing so much as faithfully summarising the only thing on the web that answered the question. That is the failure mode worth taking away, because it survives better models. Google shipped restrictions on satire and user-generated content in Overviews, and limits on where they appear at all.

Liz Reid, Google Search

One area we identified was our ability to interpret nonsensical queries and satirical content. Let's take a look at an example: "How many rocks should I eat?" Prior to these screenshots going viral, practically no one asked Google that question. [...] This is what is often called a "data void" or "information gap," where there's a limited amount of high quality content about a topic. [...] In other examples, we saw AI Overviews that featured sarcastic or troll-y content from discussion forums. Forums are often a great source of authentic, first-hand information, but in some cases can lead to less-than-helpful advice, like using glue to get cheese to stick to pizza.

AI Incident Database 10 May 2024

Incident 983: Scammers Reportedly Used AI Voice Clone and YouTube Footage to Impersonate WPP CEO in Unsuccessful... →

Someone set up a WhatsApp account using a public photograph of WPP's chief executive, used it to arrange a Microsoft Teams meeting, and ran the meeting with a voice clone of another senior executive over YouTube footage of them, typing in the chat window as Read himself while staying off camera. The target, an agency leader, was asked to set up a new business, and the attempt failed. Read wrote to WPP's leadership about it in May 2024 and the line above is from that message. Filed next to the Arup case because the two sit either side of the same threshold: one worked and one did not, and the difference was not the technology.

Mark Read, WPP

We all need to be vigilant to the techniques that go beyond emails to take advantage of virtual meetings, AI and deepfakes.

The Register 26 April 2024

Man arrested for →

A 42-second clip circulated in January 2024 appearing to be Pikesville High School's principal making racist and antisemitic remarks about students and staff. He was placed on leave, and the school required a police presence. Analysis by the FBI and by Hany Farid at Berkeley found the audio had been generated. In April police arrested the school's athletic director, Dazhon Darien, who was facing an investigation into school funds at the time, and charged him with witness retaliation, stalking, theft and disrupting school operations. Every other fraud on this page targets money. This one is here because it targets a person's standing in their own community, it took three months and a federal lab to disprove, and the tools cost nothing.

Robert McCullough, Baltimore County Police

On January 17, 2024, the Baltimore County Police Department became aware of a voice recording being circulated on social media. It was alleged the voice captured on the audio file belong to Mr Eric Eiswert, the Principal at the Pikesville High School. We now have conclusive evidence that the recording was not authentic.

Axios 17 April 2024

Amazon says shoppers are not being surveiled in Just Walk Out locations →

Just Walk Out let a shopper take things off a shelf and leave, with the bill arriving later. In April 2024 The Information reported that more than a thousand people, largely in India, were reviewing transactions, and that in 2022 around 700 of every 1,000 sales went through human review. Amazon disputes the framing, and its response is quoted here rather than the report, because on this case the dispute is the substance: nobody argues that people were involved, the argument is about how many and whether they were labelling recordings to train the system or completing the transaction. Amazon removed Just Walk Out from its Fresh grocery stores the same month, replacing it with smart trolleys, while continuing to sell it to third parties. Filed alongside Builder.ai because the shape recurs: what a demonstration shows and what the running system costs are different questions, and only one of them is usually published.

Amazon

Amazon stated that the notion of human reviewers watching live shoppers was "completely not true," and that workers in India watch some videos "after the fact" to teach machines, with "way less than 1,000" people helping in "a small percentage of cases".

AI Incident Database 10 April 2024

Incident 1100: Reported Audio Deepfake Impersonating CEO Karim Toubba Targets LastPass Employee via WhatsApp →

April 2024. The calls and voice messages came over WhatsApp, outside every channel LastPass uses internally, and pressed for urgency. The employee ignored them and told the security team, and LastPass published the attempt the same day rather than sitting on it. It is the third failed attempt on this page, with Ferrari and WPP, and the three share a shape worth copying: in each one the voice was convincing and the wrapper around it was not. A finance request arriving on WhatsApp is a fact about the process, and a process fact is checkable in a way that a voice no longer is.

A LastPass employee was reportedly targeted by an audio deepfake impersonating CEO Karim Toubba via WhatsApp. The message used voice-cloning AI in a social engineering attempt to create urgency and bypass security protocols. The employee recognized red flags, reported the incident, and no breach occurred.

arXiv 2 April 2024

Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack →

Crescendo never asks for the thing it wants. It opens with a harmless question, then asks the model to expand on its own previous answer, and repeats, each step a short distance from the last, until the conversation has arrived somewhere the model would have refused outright at the start. The Microsoft team behind it reported high success rates against ChatGPT, Gemini Pro and Ultra, Llama 2 and Anthropic's chat models, and automated it. Its significance for anyone shipping a product is that every input in the sequence passes a filter reading one message at a time. The attack is in the trajectory, and that is not what most guardrails look at. Published April 2024, accepted at USENIX Security 2025.

Mark Russinovich, Ahmed Salem and Ronen Eldan

Large Language Models (LLMs) have risen significantly in popularity and are increasingly being adopted across multiple applications. These LLMs are heavily aligned to resist engaging in illegal or unethical topics as a means to avoid contributing to responsible AI harms.

Anthropic 2 April 2024

Many →

A million-token context window is a feature everybody wanted and it is also room for hundreds of fabricated exchanges in which the assistant answers questions it should refuse. Put enough of them in front of the real question and the model treats answering as the pattern it is being asked to continue. Anthropic published this on its own models in April 2024, having briefed other developers first, and found that fine-tuning only raised the number of examples needed, while classifying and rewriting the prompt before it reached the model cut the success rate from 61 percent to 2. That is the same conclusion the Microsoft and HiddenLayer cases reach: what holds is a check outside the model, and a capability the users asked for is what made the attack possible.

We investigated a "jailbreaking" technique -- a method that can be used to evade the safety guardrails put in place by the developers of large language models (LLMs). The technique, which we call "many-shot jailbreaking", is effective on Anthropic's own models, as well as those produced by other AI companies. [...] The technique takes advantage of a feature of LLMs that has grown dramatically in the last year: the context window.

The Markup 29 March 2024

NYC’s AI Chatbot Tells Businesses to Break the Law →

New York City launched the MyCity chatbot in October 2023 to help small business owners work out what the rules are. The Markup tested it five months later and found it confidently describing illegal conduct as permitted, in several languages, on housing discrimination, tips, scheduling and funeral pricing. One detail is worth keeping: when a single reporter asked whether landlords must accept housing vouchers, the bot once said yes, but when ten staffers asked the same question it told all ten no. A wrong answer that is not reliably wrong is also not reliably testable. The city left the bot up, labelled as a beta.

Colin Lecher, The Markup

If you're a landlord wondering which tenants you have to accept, for example, you might pose a question like, "are buildings required to accept section 8 vouchers?" [...] In testing by The Markup, the bot said no, landlords do not need to accept these tenants. Except, in New York City, it's illegal for landlords to discriminate by source of income [...] The bot said it was fine to take workers' tips (wrong, although they sometimes can count tips toward minimum wage requirements) and that there were no regulations on informing staff about scheduling changes (also wrong).

NBC News 27 February 2024

A Willy Wonka-inspired experience ‘scam’ was so bad that people called the cops →

Families paid around 35 pounds a ticket for the event the promotional images showed, and arrived at a sparsely decorated warehouse where actors had been handed an incoherent script and improvised. Police were called. The organiser apologised and refunded ticket holders. The images themselves are the case: generated from a description of a thing that did not exist, complete with invented words on the signage, and good enough that hundreds of people made travel plans around them. What used to require building the thing, or at least photographing something, now requires a sentence. It is the cheapest and most ordinary version of a problem the deepfake cases further down this page describe at the other end of the scale.

Willy's Chocolate Experience was an unlicensed event based on Charlie and the Chocolate Factory that took place in Glasgow, Scotland, in February 2024, and gained worldwide notoriety for its poor execution. [...] The event was advertised as an immersive and interactive family experience, illustrated on a promotional website with "dreamlike" AI-generated imagery.

Google 23 February 2024

Gemini image generation got it wrong. We'll do better. →

Gemini's image generation launched at the start of February 2024 and was withdrawn three weeks later, after it returned racially varied images for prompts where the history is specific, including German soldiers of 1943 and America's founders, and refused a run of ordinary requests. Google paused image generation of people entirely while it reworked the feature. The post is worth reading as an engineering account rather than an apology: a correction applied globally to avoid one failure mode produced a second one, and a safety layer drifted more conservative than anyone had specified. Both are properties of a system tuned after the fact rather than of the model underneath it.

Prabhakar Raghavan, Google

It's clear that this feature missed the mark. Some of the images generated are inaccurate or even offensive. [...] So what went wrong? In short, two things. First, our tuning to ensure that Gemini showed a range of people failed to account for cases that should clearly not show a range. And second, over time, the model became way more cautious than we intended and refused to answer certain prompts entirely -- wrongly interpreting some very anodyne prompts as sensitive. These two things led the model to overcompensate in some cases, and be over-conservative in others, leading to images that were embarrassing and wrong.

Science Integrity Digest 15 February 2024

The rat with the big balls and the enormous penis – how Frontiers published a paper with botched AI →

A review article on stem cells appeared in a Frontiers journal in February 2024 illustrated with Midjourney figures whose labels are not words: protemns for proteins, dissilced, testtomcels. The anatomy in the main figure is not anatomy. The authors had disclosed that the images were generated, one of the two reviewers had flagged problems with the figures, and it was published anyway; Frontiers retracted it three days later. The interesting failure is not the generator, which did what it was asked. It is that a disclosure was made, a reviewer raised it, and the process moved past both, which is what a check looks like when nobody is required to act on it.

Elisabeth Bik

The figures in the paper were generated by the AI tool Midjourney, which generated some pretty, but nonsensical, illustrations with unreadable text. [...] It appears that neither the editor nor the two peer reviewers looked at the figures at all. The paper was peer-reviewed within a couple of weeks and published two days ago. [...] The authors disclose that the figures were generated by Midjourney, but the images are -- ahem -- anatomically and scientifically incorrect.