While exploring libraries and frameworks that leverage LLMs for user-facing applications, we came across the Vanna.AI library -- which offers a text-to-SQL interface for users -- where we discovered CVE-2024-5565, a remote code execution vulnerability via prompt injection techniques. This is an example of a "Prompt Injection" attack type, that affects LLMs, and how to override "pre-prompting" safeguards used with LLMs.
Vanna lets somebody ask a database a question in English. It turns the question into SQL, runs it, and then, to draw the chart, asks the model to write Python and executes that too. JFrog found in June 2024 that a question could carry instructions that changed what the charting code did, so a text box wired to a read-only reporting tool became arbitrary code execution on the server, rated 8.1. The instruction the library gave the model not to do this was itself just text in the same prompt. This is the pattern to check for in your own product: not whether the model can be made to say something wrong, but whether anything downstream of it runs what it says.
