What went wrong when people put AI to work.

Kusari 1 June 2026

AI Coding Assistants in 2026: 4× Faster, 10× Riskier and The Hidden Security Cost →

Five years after the NYU study that first measured this, the numbers have not improved and the volume has. The gap in the first sentence is the whole finding: almost everyone is generating code with an assistant and well under half are using one to review it, so output has accelerated past the review capacity that was already the bottleneck. The multipliers in the second are worth reading carefully, because they are ratios rather than absolutes and human code is not the clean baseline they imply. What they do establish is direction: more code, less of it read, and a measurable tilt toward the specific classes of bug that get exploited.

Nx 1 September 2025

S1ngularity - What Happened, How We Responded, What We Learned →

Nx is a build system with around four million weekly downloads. The way in was an unsanitised pull request title in a GitHub Actions workflow running with pull_request_target permissions, which leaked the npm publishing token. What makes this the first of its kind is what the payload then did: rather than search for secrets itself, it looked for AI coding assistants already installed and signed in on the developer's machine and asked them to do the reconnaissance, then pushed the results to a public repository created inside the victim's own GitHub account. A second wave two days later used the stolen credentials to make private repositories public. Nx moved to npm Trusted Publishers and manual release approval, and published this account itself.

Check Point Research 5 August 2025

Critical RCE Vulnerability in Cursor IDE Exposed →

Cursor asks before it will run a Model Context Protocol server, which is how an AI editor reaches tools and data. Check Point found in July 2025 that the approval attached to the fact of the server rather than to what it does: once a teammate had accepted a harmless entry, anyone who could commit to the repository could rewrite that entry to run any command, and it would run silently on every colleague's machine from then on. Disclosed 16 July, fixed in Cursor 1.3 on 29 July by re-prompting on any change to the configuration, down to an added space. It sits alongside a second Cursor flaw disclosed the same week and is the cleanest example of the new class of problem here: consent given once to an agent's tool, and the tool changing underneath it.

404 Media 23 July 2025

Hacker Plants Computer 'Wiping' Commands in Amazon's AI Coding Agent →

The person who did it told 404 Media they submitted an ordinary pull request to the Amazon Q extension's GitHub repository, were granted access, and put a destructive instruction into the agent's own prompt. Amazon shipped it to users in version 1.84.0 of the VS Code extension before pulling it. The wipe command was unlikely to work as written, and the person says the point was to show that the review process would not catch it. What is new here is the payload's form: not code that deletes files, but English that asks an agent with filesystem access to, which no dependency scanner is looking for.

Daniel Stenberg 14 July 2025

Death by a thousand slops →

curl is one of the most widely deployed pieces of software in the world, and its security team is a handful of volunteers. Daniel Stenberg, who wrote it, has been publishing the arithmetic of what generative AI has done to their inbox since January 2024: reports that read as competent security research, cite real-looking functions and describe vulnerabilities that do not exist, each of which still has to be read by a human before it can be dismissed. By July 2025 a fifth of all submissions were slop and one in twenty was a real vulnerability. The curl bug bounty had paid out more than 90,000 US dollars across 81 genuine findings since 2019. Stenberg spent the rest of the year weighing whether the money was now what was drawing the noise.

Matt Palmer 29 May 2025

Statement on CVE-2025 →

Lovable builds an application from a description, with Supabase behind it. Supabase enforces access at the database through row level security policies, so a generated app whose policies do not match its own login logic will happily serve one user's rows to anybody who edits the request, because the only thing stopping them was the front end. Palmer scanned 1,645 Lovable apps and found 170 of them, around one in ten, exposing data across 303 endpoints. He disclosed it, the 45-day window passed without a fix he considered adequate, and he published as CVE-2025-48757 on 29 May 2025 with a 9.3 severity. The sentence to sit with is the last one: a check that a policy exists is not a check that the policy is right, and it reads green either way.

In March 2025 a founder announced a paid lead-generation product built with an AI editor and no hand-written code. Two days later he posted that it was under attack: the API keys were in the front-end bundle and had been exhausted, the subscription check was a JavaScript condition anyone could step past, and the database was accepting writes from strangers. He took the product down. There is no novel vulnerability here and that is the point: an assistant asked for a paywall produces something that looks like a paywall, because nothing in the request said the check had to happen on a server the user does not control. Every failure in this case is one an experienced engineer would have caught in review, and the working product is what made the review feel unnecessary.

JFrog 27 June 2024

When Prompts Go Rogue: Analyzing a Prompt Injection Code Execution in Vanna.AI →

Vanna lets somebody ask a database a question in English. It turns the question into SQL, runs it, and then, to draw the chart, asks the model to write Python and executes that too. JFrog found in June 2024 that a question could carry instructions that changed what the charting code did, so a text box wired to a read-only reporting tool became arbitrary code execution on the server, rated 8.1. The instruction the library gave the model not to do this was itself just text in the same prompt. This is the pattern to check for in your own product: not whether the model can be made to say something wrong, but whether anything downstream of it runs what it says.

arXiv 12 June 2024

We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs →

When a coding model does not know the library for a job it will often name one anyway, and the name is plausible. Across 576,000 generated Python and JavaScript samples from sixteen models, this study found 5.2 percent of imports from commercial models and 21.7 percent from open-source ones pointed at packages that do not exist, amounting to 205,474 distinct invented names. The finding that makes it a supply chain problem rather than a nuisance is the repetition: re-run the same prompt ten times and 43 percent of the invented names come back every single time. That makes them predictable enough to register in advance, which is the attack now called slopsquatting: the attacker does not need to guess what a developer will type, only what their assistant will invent. Presented at USENIX Security 2025.

The Register 6 December 2022

Stack Overflow bans ChatGPT as →

ChatGPT was five days old. Stack Overflow's moderators banned answers written with it, in a policy statement headed simply "ChatGPT is banned", on the grounds that the answers were plausible enough to post and wrong often enough to be worse than nothing, and that checking them was falling on volunteers. It is the earliest institutional response to generative AI anywhere and the argument has not changed since: the cost of a confident wrong answer is not borne by the person who generated it. Three years later the same argument closed curl's bug bounty. Stack Overflow's own question volume has fallen sharply over the period, which is the part nobody at the time predicted.

arXiv 16 December 2021

Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions →

The first systematic security study of an AI coding assistant, run at NYU while GitHub Copilot was still in preview. The team built scenarios where MITRE's twenty-five most dangerous software weaknesses were possible, let Copilot complete them, and analysed the output: about two in five programs carried a vulnerability. The paper is careful about what that does and does not mean, since the same figure would not be flattering for human code either, and its more useful finding is the second one: the security of a completion depends heavily on how the surrounding code is written, so a developer whose own file is already sloppy gets worse suggestions. Published in 2021 and cited in almost every argument about this since.