-
Simon Willison named prompt injection in 2022 and has tracked it since. This post is the shortest statement of why it keeps happening: a language model cannot reliably tell an instruction from its operator apart from an instruction sitting inside the content it was asked to read, so any system that holds private data, reads untrusted input and can send anything outwards can be made to carry the first out through the third. Any two of the three are manageable. Most of the vulnerabilities filed on this page are that sentence happening to a specific product.
From Simon Willison
If you are a user of LLM systems that use tools (you can call them "AI agents" if you like) it is critically important that you understand the risk of combining tools with the following three characteristics. Failing to understand this can let an attacker steal your data.
The lethal trifecta of capabilities is:
Access to your private data [...]
Exposure to untrusted content [...]
The ability to externally communicate in a way that could be used to steal your data
eChai Partner Brands
The partnership →
