Wrong Gone AI · eChai

Malwarebytes 4 August 2026

Online backlash ends in Google rolling back Google Earth AI tool after a day →

Google switched on Gemini image generation inside Google Earth on 30 July 2026, for everyone, pitched at students bringing history to life and estate agents making site plans. The problem is the base layer: generating on top of real satellite imagery means the buildings, the coastline and the roads are correct, so the output is a fabrication with a true map underneath it. The Dutch open source researcher quoted here tested exactly the cases you would expect a safety filter to catch, and reported that none of them were refused; NPR and Futurism showed the same. Google rolled the feature back worldwide in under twenty-four hours. The fastest withdrawal on this page, and the failure is not the generator but the decision to point it at a product people use to check whether something is really there.

Henk van Ess

I tried refugees at the Mexican border, a nuclear plant in Iran, a crash in Amsterdam, a hospital with a bomb crater in Gaza. Nothing was refused.

forbes.com Google Pulls Earth AI Tool After It Faked A Nuclear Plant In Iran

iPleaders 6 July 2026

AI-hallucinated case law: how fake citations are getting lawyers sanctioned in India (and how to avoid it) →

India's courts said this early. The Delhi High Court's Louboutin ruling is from August 2023, two months after Mata v. Avianca in New York, and it has been quoted in Indian courtrooms ever since. What followed anyway: a tax tribunal in Bengaluru recalling an order of roughly 669 crore rupees over four citations that do not exist; the Bombay High Court quashing a faceless tax assessment of about 27.91 crore in October 2025 because it rested on three invented precedents; and a 50,000 rupee cost order from the same court in January 2026, where what gave the submission away was not the citation but the writing, green tick marks copied out of a chat window and phrasing that repeated itself. The consequence that matters commercially is in the second of those: the order collapses, and the matter reopens.

Delhi High Court

In Christian Louboutin SAS v. M/s The Shoe Boutique-Shutiq (22 August 2023), the Delhi High Court, dealing with a trademark dispute in which a party had leaned on ChatGPT, held plainly that "the responses generated by ChatGPT cannot be the basis of adjudication of legal or factual issues in a court of law." The Court cautioned against "fictional case laws" and "imaginative data," and placed AI firmly in the category of a tool for preliminary understanding, not a source of authority.

MediaNama 3 July 2026

10 cases that show Indian courts have an AI hallucination problem →

On 2 July 2026 a bench of Justices P S Narasimha and Alok Aradhe set aside orders of the National Company Law Tribunal and its appellate tribunal in the Essel Infraprojects insolvency matter. Of the six judgments the tribunal had relied on, three do not exist and three real ones carried invented paragraphs or the wrong case title. The detail that makes this different from every lawyer-sanctioned case elsewhere on this page is in Jammu and Kashmir Bank's affidavit: counsel had cited none of the six. The tribunal had found them itself, and the appellate tribunal did not catch them either. The court's comparison is to Bhopal. It directed the Bar Council of India to frame disciplinary norms.

Supreme Court of India

The SC called it misconduct for an advocate and a serious lapse for a judge, directed the Bar Council of India (BCI) to frame disciplinary norms, and likened hallucinated case law to "the release of methyl isocyanate in the province of law and justice."

livelaw.in Phantom Precedents: The Rise Of AI-Generated Case Law In Indian Courts

Kusari 1 June 2026

AI Coding Assistants in 2026: 4× Faster, 10× Riskier and The Hidden Security Cost →

Five years after the NYU study that first measured this, the numbers have not improved and the volume has. The gap in the first sentence is the whole finding: almost everyone is generating code with an assistant and well under half are using one to review it, so output has accelerated past the review capacity that was already the bottleneck. The multipliers in the second are worth reading carefully, because they are ratios rather than absolutes and human code is not the clean baseline they imply. What they do establish is direction: more code, less of it read, and a measurable tilt toward the specific classes of bug that get exploited.

85% of respondents use AI coding assistants -- but only 38% apply AI to support code review in pull requests. [...] AI-generated code produced 1.75x more logic and correctness errors, 1.57x more security findings, and was 2.74x more likely to introduce cross-site scripting (XSS) vulnerabilities.

securityboulevard.com Your Coding Assistant Is Shipping Security Vulnerabilities

Forbes 29 May 2026

A Federal Judge, A 1967 Law And A Billion Rejected Job Applications →

Derek Mobley says he was rejected from more than a hundred jobs screened through Workday's tools, and alleges the screening discriminated on race, age and disability. The court granted preliminary certification of an age collective in May 2025; around 14,000 people opted in before the window closed in March 2026, and the claims under Title VII, the ADEA, the ADA and California's FEHA have survived dismissal. The finding that matters beyond this case is in the quote: the court let the claims proceed against the VENDOR, on the basis that supplying the screening makes it an agent of the employer. Every case elsewhere on this page about a system deciding something about a person was brought against the organisation that deployed it. This one is aimed at the company that built it.

Mobley v. Workday is a federal class and collective action in the Northern District of California challenging the AI-driven applicant screening tools Workday provides to thousands of employers. [...] The Court permitted the claims to proceed on the basis that the operative complaint adequately alleged that Workday is an agent of its client-employers and therefore falls within the definition of "employer" under Title VII, the ADEA, and the ADA.

oecd.ai 12 May 2026

Ontario Medical AI Scribes Found to Produce Dangerous Errors and Hallucinations →

An AI scribe listens to a consultation and writes the note that goes in the chart. Ontario's auditor general examined the tools approved for use by doctors in the province and reported that most produced inaccurate, incomplete or fabricated notes, with serious errors found across twenty programmes during procurement, and warned that a wrong note can produce a treatment plan that harms the patient. The specific failure is not mishearing: scribes have been found inserting diagnoses and examination findings that were never discussed. Read beside the Whisper case from 2024. That was one model invented sentences in the silences; this is a provincial health system having bought twenty products built on that class of model, and checking afterwards.

OECD AI Incidents Monitor

Ontario Medical AI Scribes Found to Produce Dangerous Errors and Hallucinations

Microsoft Security 8 May 2026

When prompts become shells: RCE vulnerabilities in AI agent frameworks →

Microsoft's own research team went after the layer under the agents rather than the agents themselves, and found two ways to turn a prompt into code execution in Semantic Kernel: CVE-2026-26030 through the in-memory vector store, and CVE-2026-25592 an arbitrary file write through the Python sessions plugin. One prompt was enough to launch a process on the machine running the agent, with no browser exploit, no attachment and no memory corruption anywhere in the chain. Both are patched. The reason it belongs on this page rather than in a bulletin is the sentence quoted: everything on this page that reads as a product's own flaw is increasingly a flaw in the four or five frameworks every product is built on.

To build powerful applications, developers rely heavily on frameworks like Semantic Kernel, LangChain, and CrewAI. These frameworks act as the operating system for AI agents, abstracting away complex model orchestration. But this convenience comes with a hidden cost: because these frameworks act as a ubiquitous foundational layer, a single vulnerability in how they map AI model outputs to system tools carries systemic risk.

Zenity | Secure AI Agents Everywhere 28 April 2026

AI Agent Destroys Production Database in 9 Seconds →

The agent was doing a routine job in STAGING. It hit a credential mismatch, decided by itself that the fix was to delete a Railway volume, went looking for a token, and found one in an unrelated file created for managing custom domains. That token had blanket authority across Railway's whole API, including volumeDelete. There was no confirmation step and no environment scoping. Car rental counters opened on the Saturday morning with no reservations, no payment records and no vehicle assignments; Railway reconstructed the data from earlier backups. Asked to explain itself, the agent listed the rules it had broken, including the instruction never to run a destructive command unasked. Read it beside the Replit case nine months earlier: same shape, better model, and the thing that failed was not the model but a token nobody had scoped.

Zenity Labs

On April 25th, a Cursor AI coding agent running Anthropic's Claude Opus 4.6 [...] deleted the production database for PocketOS, a software platform used by car rental businesses across the country to manage their entire operations. The deletion took 9 seconds. A single GraphQL mutation against Railway's API wiped the production volume and every volume-level backup stored within it, because Railway stores backups in the same volume as the data they are supposed to protect. [...] The Cursor agent was not compromised by an attacker, it was not manipulated through prompt injection, and it was not running malicious code. It was trying to accomplish the goal it had been given, encountered an obstacle, and made an autonomous decision about how to remove that obstacle.

theregister.com Cursor-Opus agent snuffs out startup's production database

Wharton AI & Analytics Initiative 14 April 2026

Two Early 2026 AI Exposures: Lessons for the Future of AI and Data Governance →

Sears' support chatbots and voice agents had been recording everything, which is the ordinary thing to do when you intend to improve a model on it. Three databases holding the result sat open on the internet with no password: names, phone numbers, home addresses, appliance and repair details, and 1.4 million audio recordings, some of which kept running after the call and picked up the room. They were closed once Fowler reported them. The last sentence quoted is the one that generalises: the misconfiguration is decades old, and what is new is that a support system now keeps a verbatim transcript and a voice print of every conversation it has, so the same open door leaks something categorically worse than a customer list.

Wharton Accountable AI Lab

cybersecurity researcher Jeremiah Fowler discovered three completely unprotected databases -- no passwords, no encryption -- containing massive amounts of data from Sears' customer-facing AI chatbots and voice agents. These included 3.7 million chat-log transcripts, 1.4 million audio recordings (some running hours long and even capturing household noise after the calls ended), and over 4 TB of plaintext data. [...] This is a classic "left the door open" misconfiguration -- a familiar data security failure -- but amplified by the practice of logging every interaction for training or quality purposes.

Live Law 17 March 2026

Phantom Precedents: The Rise Of AI-Generated Case Law In Indian Courts →

Two of the Indian cases that led to the Supreme Court's ruling. In February 2026 a bench headed by Chief Justice Surya Kant raised AI-drafted petitions being filed without anybody checking them, and Justice Nagarathna said she had been handed a case that does not exist. Five months earlier a Delhi High Court petition had quoted paragraphs 73 and 74 of a 1972 Supreme Court judgment that runs to 27 paragraphs, and was withdrawn once opposing counsel checked. That is the harder version of the failure and it recurs across these cases: not an invented case, which a search will disprove, but invented text inside a real one, which only somebody who opens the judgment will catch.

Justice B.V. Nagarathna specifically highlighted that she encountered a reference to a fictitious case titled Mercy v. Mankind while hearing a Public Interest Litigation (PIL). [...] in September 2025, the Delhi High Court saw a petition withdrawn in embarrassment after the opposing counsel exposed the citations as completely fabricated. The petition even went so far as to invent phantom paragraphs from the landmark "Raj Narain v. Indira Nehru Gandhi, (1972) 3 SCC 850" judgment, heavily quoting paragraphs 73 and 74 from a ruling that only contains 27 paragraphs in reality.

ABA Journal 2 March 2026

Federal judge hands down $110K penalty against 2 lawyers for AI errors in court documents →

Four siblings fighting over a family winery after their mother's death, and a 12 million dollar claim. Over five months and three separate briefs on cross-motions for summary judgment, the plaintiffs' lawyers filed fifteen cases that do not exist and eight quotations attributed to real decisions that do not contain them. The judge called it a notorious outlier in degree and volume. At roughly 110,000 US dollars it is the largest penalty for fabricated citations in any United States court to date, three years after Mata v. Avianca set the going rate at 5,000. The tracking database now holds around 1,500 decisions worldwide where a court has had to respond to hallucinated material, and the sanctions have reached bar suspensions.

A federal magistrate judge in the District of Oregon sanctioned two lawyers a combined $110,204.38 for filing summary-judgment briefs that cited 15 nonexistent cases and 8 fabricated quotations generated with AI.

Malwarebytes 9 February 2026

AI chat app leak exposes 300 million messages tied to 25 million users →

A Firebase database left open with no authentication, holding the complete chat histories of 25 million people along with their settings and the models they had used. What makes it worse than the usual misconfiguration is what people take to a chat app they believe is private, and the reporting is clear that the exposed conversations included some of the most sensitive things a person can type. The second half of the quote is the part founders should sit with: this is a wrapper. It holds no model of its own and its whole product is a front end onto three other companies' APIs, which means the safety work those three did was never the thing protecting these users. The weakest link in an AI product is routinely the ordinary web infrastructure around it.

Pieter Arntz, Malwarebytes

An independent security researcher uncovered a major data breach affecting Chat & Ask AI, one of the most popular AI chat apps on Google Play and Apple App Store, with more than 50 million users. The researcher claims to have accessed 300 million messages from over 25 million users due to an exposed database. [...] Behind the scenes, Chat & Ask AI is a "wrapper" app that plugs into various large language models (LLMs) from other companies, including OpenAI's ChatGPT, Anthropic's Claude, and Google's Gemini.

Anthropic 13 November 2025

Disrupting the first reported AI-orchestrated cyber espionage campaign →

The most consequential jailbreak on this page, and the technique is the oldest one here: tell the model it works for a security firm doing authorised testing, then hand it the work in pieces small enough that no single request looks like an attack. Anthropic detected it in mid-September 2025, spent ten days mapping it, banned accounts as it found them, notified the organisations and went to the authorities. The targets were technology companies, banks, chemical manufacturers and government agencies, and the model did the reconnaissance, the vulnerability discovery, the exploitation and the extraction itself. Every earlier case in this category is a researcher proving a technique in a lab. This is that technique used at national scale, disclosed by the company whose model it was.

The threat actor -- whom we assess with high confidence was a Chinese state-sponsored group -- manipulated our Claude Code tool into attempting infiltration into roughly thirty global targets and succeeded in a small number of cases. [...] We believe this is the first documented case of a large-scale cyberattack executed without substantial human intervention. [...] At this point they had to convince Claude -- which is extensively trained to avoid harmful behaviors -- to engage in the attack. They did so by jailbreaking it [...] They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a legitimate cybersecurity firm, and was being used in defensive testing.

incidentdatabase.ai Incident 1263: Chinese State-Linked Operator (GTG-1002) Uses Claude Code for Autonomous Cyber Espionage

The National Law Review 1 November 2025

TAKE NOTE- New Wave of Privacy Litigation Targets AI Notetaker, Otter.ai →

An AI notetaker joins a call as a participant, and the person who invited it has an account and agreed to terms. Everybody else on the call has neither. The class action filed in California in August 2025 turns on that gap, under federal and state wiretap law, and the named plaintiff is not an Otter user at all: he says he was recorded because somebody else's tool was in the meeting. Related complaints raise voiceprints under Illinois biometric law. The allegations are untested. They are on this page because the practice they describe is now the default in most companies, and the question of who is supposed to collect consent from the other eight people on a call has no settled answer.

In Brewer v. Otter.ai Inc., No. 5:25-cv-06911 (N.D. Cal. Aug. 15, 2025), the complaint alleges that Otter automatically joins Google Meet, Zoom, and Microsoft Teams meetings and records, accesses, reads, and learns the contents of conversations involving non-users without their consent [...] According to the allegations, Otter transmits call content to its servers in real time, uses participant conversations to train its machine-learning models, retains recordings indefinitely, and provides no meaningful disclosure to non-Otter participants.

NeuralTrust 24 October 2025

OpenAI Atlas Omnibox Prompt Injection: URLs That Become Jailbreaks →

Atlas is OpenAI's browser, and its address bar takes both a URL and an instruction. NeuralTrust found that a string starting https: and shaped like a domain, but not actually a valid URL, falls through to being treated as an instruction from the user, which is the highest trust level the agent has. So a link somebody copies from a post and pastes into the bar can carry commands rather than a destination, with the reader's own logged-in sessions behind them: NeuralTrust demonstrated sending the agent to a lookalike sign-in page and, separately, into the user's Google Drive to delete files. Disclosed 24 October 2025. It is worth reading beside the Brave and Comet work: an agentic browser widens the trust boundary to include anything a person pastes as well as anything a page contains.

NeuralTrust research shows how using crafted strings that resemble URLs, an attacker can override user intent and jailbreak agentic browsers like OpenAI Atlas.

Legit Security 8 October 2025

CamoLeak: Critical GitHub Copilot Vulnerability Leaks Private Source Code →

The same researcher who did the GitLab Duo work, five months later, on GitHub. Instructions hidden in a pull request comment were read by Copilot Chat when a maintainer asked it about the repository, and the answer to how the data got out is the elegant part: GitHub proxies remote images through its own domain, Camo, so an image URL built by the injected instruction is served from GitHub itself and passes GitHub's own content security policy. Private source code, AWS keys and unpublished vulnerability reports all went out that way. GitHub's fix was to stop Copilot Chat rendering images at all, which is the same fix Google reached on Bard two years earlier and the same one GitLab reached in between.

Omer Mayraz, Legit Security

In June 2025, I found a critical vulnerability in GitHub Copilot Chat (CVSS 9.6) that allowed silent exfiltration of secrets and source code from private repos, and gave me full control over Copilot's responses, including suggesting malicious code or links. The attack combined a novel CSP bypass using GitHub's own infrastructure with remote prompt injection. I reported it via HackerOne, and GitHub fixed it by disabling image rendering in Copilot Chat completely.

The Register 6 October 2025

Deloitte refunds Australian government over AI in report →

Australia's Department of Employment and Workplace Relations paid Deloitte around 440,000 Australian dollars to review the Targeted Compliance Framework, the automated system that penalises welfare recipients. The report was published in August 2025. A University of Sydney academic, Chris Rudge, then found citations to papers that do not exist, including several attributed to a real law professor, and a fabricated quotation from a Federal Court judgment in the robodebt case. Deloitte issued a corrected version stripping more than a dozen references, added a disclosure of the AI toolchain that had not been in the original, and refunded the final instalment, about 97,000 Australian dollars. The department says the findings and recommendations are unchanged.

Deloitte has agreed to refund part of an Australian government contract after admitting it used generative AI to produce a report riddled with fake citations, phantom footnotes, and even a made-up quote from a Federal Court judgment. [...] Now the new version contains a disclosure in its methodology section: Deloitte used "a generative AI large language model (Azure OpenAI GPT-4o) based tool chain licensed by DEWR and hosted on DEWR's Azure tenancy" to fill "traceability and documentation gaps."

Reason 25 September 2025

Court Throws Out Case After Finding Plaintiffs Submitted Deepfake Videos and Altered Images →

Every other fabrication case on this page is a citation: a judgment that does not exist, cited by somebody who did not check. This is the first reported one where the fabricated thing was EVIDENCE. Self-represented plaintiffs filed video exhibits presenting a witness giving testimony she never gave, and the court identified them itself: seven to ten seconds looping, a monotone voice over digitally moved lips, and Ring images with a pixelated grey background behind a subject in crisp colour. The judge compared them against real footage of the same person and found the accent, cadence, word choice and gestures were somebody else's. Decided 9 September 2025, dismissed with prejudice. What makes it consequential beyond the case is the second-order problem the courts now name the liar's dividend: once juries know this exists, real recordings can be denied.

Judge Victoria Kolakowski, Superior Court of California, Alameda County

The Court finds that Plaintiffs violated section 128.7(b) of the Code of Civil Procedure by submitting fabricated evidence in support of their motion for summary judgment. [...] The Court finds that exhibits 6A and 6C are products of GenAI and do not capture the actual speech and image of Geri Haas. In other words, these exhibits are deepfakes. [...] a terminating sanction serves the appropriate deterrent effect of showing the public that the Court has zero tolerance with attempting to pass deepfakes as evidence.

ediscoverytoday.com Deepfake Videos and Images Lead to Terminating Sanctions

Noma Security 25 September 2025

ForcedLeak: AI agent risks exposed in Salesforce Agentforce →

Web-to-Lead is the Salesforce form on a company's own website that turns an enquiry into a CRM record. Anyone on the internet can fill one in. Noma Labs put instructions in the Description field, and when an employee later asked Agentforce about their leads the agent read them and obeyed. Getting the data out needed one more thing, because Salesforce only allows the agent to reach allowlisted domains: Noma found a domain on that list that had expired, and bought it for five dollars. Reported on 28 July 2025; Salesforce re-secured the domain and shipped Trusted URL enforcement on 8 September. The expired allowlist entry is the part worth acting on, because every company with an agent has a list like that and nobody is watching it for renewals.

Sasi Levi, Noma Labs

This research outlines how Noma Labs discovered ForcedLeak, a critical severity (CVSS 9.4) vulnerability chain in Salesforce Agentforce that could enable external attackers to exfiltrate sensitive CRM data through an indirect prompt injection attack. [...] The LLM, operating as a straightforward execution engine, lacked the ability to distinguish between legitimate data loaded into its context and malicious instructions that should only be executed from trusted sources, resulting in critical sensitive data leakage.

Simon Willison 19 September 2025

The Hidden Risk in Notion 3.0 AI Agents: Web Search Tool Abuse for Data Exfiltration →

Notion shipped agents on 18 September 2025 with access to a workspace, the ability to read uploaded files, and a web search tool. That is the lethal trifecta assembled in one product, and CodeIntegrity demonstrated it the next day: a PDF presented as a customer feedback report, carrying white-on-white instructions dressed up as an internal procedure complete with a pre-authorised and safe reassurance. A user uploads it and asks for a summary; the agent reads the client list and sends names, companies and revenue out through the search tool's own network requests. Notion shipped mitigations afterwards. The reason the search tool is the exfiltration path is that it is the one capability whose whole job is to make outbound requests.

Abi Raghuram reports that Notion 3.0, released yesterday, introduces new prompt injection data exfiltration vulnerabilities thanks to enabling lethal trifecta attacks. Abi's attack involves a PDF with hidden text (white text on white background) that tricks Notion's agent (in this case using Claude Sonnet 4) into doing the following.