We investigated a "jailbreaking" technique -- a method that can be used to evade the safety guardrails put in place by the developers of large language models (LLMs). The technique, which we call "many-shot jailbreaking", is effective on Anthropic's own models, as well as those produced by other AI companies. [...] The technique takes advantage of a feature of LLMs that has grown dramatically in the last year: the context window.
Curated from anthropic.com · 2 April 2024 →
A million-token context window is a feature everybody wanted and it is also room for hundreds of fabricated exchanges in which the assistant answers questions it should refuse. Put enough of them in front of the real question and the model treats answering as the pattern it is being asked to continue. Anthropic published this on its own models in April 2024, having briefed other developers first, and found that fine-tuning only raised the number of examples needed, while classifying and rewriting the prompt before it reached the model cut the success rate from 61 percent to 2. That is the same conclusion the Microsoft and HiddenLayer cases reach: what holds is a check outside the model, and a capability the users asked for is what made the attack possible.
