Gone Mad AI

What went wrong when people put AI to work.

A lawyer filed six court cases that did not exist. A coding agent dropped a production database and then wrote up what it had done. An airline was held to a refund policy its own support bot invented. We are collecting what went wrong, case by case, each one filed under the kind of failure it was and carrying the source it is drawn from. Nobody in these bylines is on eChai, and every entry links back to where it was reported.

Founder Notes →

Curated from neuraltrust.ai · 24 October 2025 →

Atlas is OpenAI's browser, and its address bar takes both a URL and an instruction. NeuralTrust found that a string starting https: and shaped like a domain, but not actually a valid URL, falls through to being treated as an instruction from the user, which is the highest trust level the agent has. So a link somebody copies from a post and pastes into the bar can carry commands rather than a destination, with the reader's own logged-in sessions behind them: NeuralTrust demonstrated sending the agent to a lookalike sign-in page and, separately, into the user's Google Drive to delete files. Disclosed 24 October 2025. It is worth reading beside the Brave and Comet work: an agentic browser widens the trust boundary to include anything a person pastes as well as anything a page contains.

NeuralTrust

NeuralTrust research shows how using crafted strings that resemble URLs, an attacker can override user intent and jailbreak agentic browsers like OpenAI Atlas.
Curated from legitsecurity.com · 8 October 2025 →

The same researcher who did the GitLab Duo work, five months later, on GitHub. Instructions hidden in a pull request comment were read by Copilot Chat when a maintainer asked it about the repository, and the answer to how the data got out is the elegant part: GitHub proxies remote images through its own domain, Camo, so an image URL built by the injected instruction is served from GitHub itself and passes GitHub's own content security policy. Private source code, AWS keys and unpublished vulnerability reports all went out that way. GitHub's fix was to stop Copilot Chat rendering images at all, which is the same fix Google reached on Bard two years earlier and the same one GitLab reached in between.

Omer Mayraz, Legit Security

In June 2025, I found a critical vulnerability in GitHub Copilot Chat (CVSS 9.6) that allowed silent exfiltration of secrets and source code from private repos, and gave me full control over Copilot's responses, including suggesting malicious code or links. The attack combined a novel CSP bypass using GitHub's own infrastructure with remote prompt injection. I reported it via HackerOne, and GitHub fixed it by disabling image rendering in Copilot Chat completely.
Curated from noma.security · 25 September 2025 →

Web-to-Lead is the Salesforce form on a company's own website that turns an enquiry into a CRM record. Anyone on the internet can fill one in. Noma Labs put instructions in the Description field, and when an employee later asked Agentforce about their leads the agent read them and obeyed. Getting the data out needed one more thing, because Salesforce only allows the agent to reach allowlisted domains: Noma found a domain on that list that had expired, and bought it for five dollars. Reported on 28 July 2025; Salesforce re-secured the domain and shipped Trusted URL enforcement on 8 September. The expired allowlist entry is the part worth acting on, because every company with an agent has a list like that and nobody is watching it for renewals.

Sasi Levi, Noma Labs

This research outlines how Noma Labs discovered ForcedLeak, a critical severity (CVSS 9.4) vulnerability chain in Salesforce Agentforce that could enable external attackers to exfiltrate sensitive CRM data through an indirect prompt injection attack. [...] The LLM, operating as a straightforward execution engine, lacked the ability to distinguish between legitimate data loaded into its context and malicious instructions that should only be executed from trusted sources, resulting in critical sensitive data leakage.
Curated from simonwillison.net · 19 September 2025 →

Notion shipped agents on 18 September 2025 with access to a workspace, the ability to read uploaded files, and a web search tool. That is the lethal trifecta assembled in one product, and CodeIntegrity demonstrated it the next day: a PDF presented as a customer feedback report, carrying white-on-white instructions dressed up as an internal procedure complete with a pre-authorised and safe reassurance. A user uploads it and asks for a summary; the agent reads the client list and sends names, companies and revenue out through the search tool's own network requests. Notion shipped mitigations afterwards. The reason the search tool is the exfiltration path is that it is the one capability whose whole job is to make outbound requests.

Simon Willison

Abi Raghuram reports that Notion 3.0, released yesterday, introduces new prompt injection data exfiltration vulnerabilities thanks to enabling lethal trifecta attacks. Abi's attack involves a PDF with hidden text (white text on white background) that tricks Notion's agent (in this case using Claude Sonnet 4) into doing the following.
Curated from thehackernews.com · 19 September 2025 →

Radware's researchers pointed the same email trick at ChatGPT's Deep Research agent with the Gmail connector switched on. The difference from every earlier version of this attack is where the data leaves from: the agent browses on OpenAI's infrastructure, so the outbound request never touches the company network and no gateway, proxy or endpoint agent sees it happen. They named it ShadowLeak, reported it on 18 June 2025, and OpenAI fixed it in early August. Their proof of concept used Gmail, but the same shape applies to any connector the agent can reach, and the list includes Box, Dropbox, GitHub, Google Drive, HubSpot, Outlook, Notion and SharePoint.

Zvika Babo, Gabi Nakibly and Maor Uziel, Radware

The attack utilizes an indirect prompt injection that can be hidden in email HTML (tiny fonts, white-on-white text, layout tricks) so the user never notices the commands, but the agent still reads and obeys them. [...] Unlike prior research that relied on client-side image rendering to trigger the leak, this attack leaks data directly from OpenAI's cloud infrastructure, making it invisible to local or enterprise defenses.
radware.com ShadowLeak: Radware Threat Intelligence
Curated from arxiv.org · 6 September 2025 →

Aim Labs reported EchoLeak to Microsoft in January 2025 and it was disclosed in June as CVE-2025-32711, rated 9.3. The attack is one email. The recipient does not have to open it, click anything or even see it: Copilot indexes the mailbox, reads the hidden instructions in the message, and then follows them the next time the employee asks Copilot an unrelated question, pulling data out of the company's own Microsoft 365 tenant. Microsoft patched it server-side and says it has no evidence of exploitation. This paper is the academic writeup of the chain; it matters because every defence in it was already deployed and each one was bypassed in turn.

Pavan Reddy and Aditya Sanjay Gujral

This paper presents an in-depth case study of EchoLeak (CVE-2025-32711), a zero-click prompt injection vulnerability in Microsoft 365 Copilot that enabled remote, unauthenticated data exfiltration via a single crafted email. By chaining multiple bypasses -- evading Microsoft's XPIA (Cross Prompt Injection Attempt) classifier, circumventing link redaction with reference-style Markdown, exploiting auto-fetched images, and abusing a Microsoft Teams proxy allowed by the content security policy -- EchoLeak achieved full privilege escalation across LLM trust boundaries without user interaction.
Curated from brave.com · 20 August 2025 →

Brave's own security team, building a browser agent of its own, tested Perplexity's Comet and found it would follow instructions planted in a Reddit comment. The chain they demonstrated is worth reading in full because nothing in it is exotic: the agent was told to open the user's Perplexity account page and read their email address, then to visit a lookalike domain (perplexity.ai. with a trailing dot, which browsers treat as a different host) to trigger a one-time password, then to open Gmail, where the user was already signed in, read the code, and post both back as a reply to the original comment. Every step used the user's existing logged-in sessions. Brave reported it to Perplexity before publishing.

Brave

On websites they control, attackers might hide instructions using white text on white backgrounds, HTML comments, or other invisible elements. Alternatively, they may inject malicious prompts into user-generated content on social media platforms such as Reddit comments or Facebook posts. [...] The attack we developed shows that traditional Web security assumptions don't hold for agentic AI, and that we need new security and privacy architectures for agentic browsing.
Curated from labs.zenity.io · 6 August 2025 →

Zenity presented this at Black Hat in August 2025. A document carrying hidden instructions is shared into somebody's Google Drive, which on Drive an attacker can do without permission. The victim never opens it. The next time they ask ChatGPT anything that causes it to search their Drive, it reads the document, follows the instructions, hunts for API keys across the rest of the account and sends them out. The same research covered Copilot Studio, Cursor with the Jira connector, Salesforce Einstein and Microsoft Copilot. Read next to the Rehberger case from 2023: the vulnerability is unchanged and the amount of data reachable through one connector has grown by two orders of magnitude.

Tamir Ishay Sharbat, Zenity Labs

Recently OpenAI added a new feature to ChatGPT called Connectors. Connectors let ChatGPT connect to third-party applications such as Google Drive, Sharepoint, Github, and more. Now your trustworthy AI companion can search files, pull live data, and give answers which are grounded in your personal business context.
Curated from simonwillison.net · 16 June 2025 →

Simon Willison named prompt injection in 2022 and has tracked it since. This post is the shortest statement of why it keeps happening: a language model cannot reliably tell an instruction from its operator apart from an instruction sitting inside the content it was asked to read, so any system that holds private data, reads untrusted input and can send anything outwards can be made to carry the first out through the third. Any two of the three are manageable. Most of the vulnerabilities filed on this page are that sentence happening to a specific product.

Simon Willison

If you are a user of LLM systems that use tools (you can call them "AI agents" if you like) it is critically important that you understand the risk of combining tools with the following three characteristics. Failing to understand this can let an attacker steal your data.

The lethal trifecta of capabilities is:

Access to your private data [...]
Exposure to untrusted content [...]
The ability to externally communicate in a way that could be used to steal your data
Curated from legitsecurity.com · 22 May 2025 →

GitLab Duo reads a project the way a colleague would: merge request descriptions, commit messages, issue comments, source files. Legit Security put instructions in each of those, hidden from a human reader with Unicode smuggling, Base16 encoding and white KaTeX text, and Duo followed every one. Because Duo renders Markdown as it streams, the injected instruction could make it emit an image tag pointing at an attacker's server with private source code encoded in the URL. Reported on 12 February 2025 and patched by restricting which HTML Duo will render. The lesson generalises past GitLab: a repository is untrusted input the moment anyone outside your team can open a merge request against it.

Legit Security research team

A hidden comment was enough to make GitLab Duo leak private source code and inject untrusted HTML into its responses. [...] a remote prompt injection vulnerability that allows attackers to steal source code from private projects, manipulate code suggestions shown to other users, and even exfiltrate confidential, undisclosed zero-day vulnerabilities -- all through GitLab Duo Chat. [...] We experimented by placing hidden prompts in [...] Every single one of these worked -- GitLab Duo responded to the hidden prompts.
Curated from promptarmor.substack.com · 20 August 2024 →

Slack AI answers questions in natural language over everything a user can see, which includes public channels. PromptArmor showed in August 2024 that an attacker who can post in any public channel, including one they create alone, can plant an instruction that Slack AI will act on when a colleague asks it something entirely different, returning content from private channels the attacker was never in. Slack's ingestion of uploaded files and Google Drive documents six days earlier had widened what was reachable. The structural point is the one that recurs on this page: a retrieval assistant inherits the reach of whoever asks it, not the reach of whoever wrote the text it is reading.

PromptArmor

In this scenario, we display how, via Slack AI, an attacker with access to Slack can exfiltrate data in private channels they are not a part of. [...] Prompt injection occurs because an LLM cannot distinguish between the "system prompt" created by a developer and the rest of the context that is appended to the query. [...] Note that Slack insider threats are already an issue (leaks from Disney, Uber, EA, Twitter, etc in the past have happened via Slack), and this just explodes the risk as now an attacker does not even need access to the private channel or data within Slack to exfiltrate it.
Curated from embracethered.com · 3 November 2023 →

Google gave Bard access to Gmail, Drive and Docs in September 2023. Johann Rehberger, who has been documenting this class of bug longer than most, put instructions inside a Google Doc and shared it with himself, which on Google Workspace an attacker can do to anybody without asking. Bard read the document, followed the instructions, and leaked the conversation by rendering a Markdown image whose URL carried the data, so the browser sent it out simply by loading the picture. Reported 19 September 2023, confirmed fixed 19 October. The image-rendering trick recurs across almost every case in this category, because an assistant that can format its answers can make the reader's own browser do the exfiltration.

Johann Rehberger

Indirect Prompt Injection attacks via Emails or Google Docs are interesting threats, because these can be delivered to users without their consent. Imagine an attacker force-sharing Google Docs with victims! When the victim searches or interacts with the attacker's document using Bard the prompt injection can kick in! [...] A common vulnerability in LLM apps is chat history exfiltration via rendering of hyperlinks and images.

eChai Partner Brands The partnership →