A team of researchers presented a paper at the International Conference on Machine Learning (ICML) this month arguing that large language models cannot be made fully secure against jailbreak attacks due to a fundamental flaw in how they process instructions2.
The core vulnerability, which the researchers call "chain-of-thought forgery," exploits how LLMs identify the role of text chunks — by style and word choice rather than by surrounding structural tags. By writing instructions in a style that mimics chain-of-thought text, an attacker can trick an LLM into behaving as if it had generated the instruction itself. Swapping tags around, such as replacing think tags with user tags, made almost no difference to how the models interpreted the text.
"There's a real probability that this is going to be a problem that's fundamentally unsolvable," said Charles Ye, an independent researcher and coauthor of the paper.
Using the technique, the researchers were able to make popular LLMs produce information they had been trained to withhold, including instructions on how to synthesize cocaine and how to sabotage a commercial aircraft's navigation system. Coauthor Jasmine Cui said GPT-5.4 gave her instructions on how to commit suicide, and that she previously persuaded a version of Anthropic's Claude to show her how to build a weapon by telling it the model was already being used by the military. OpenAI's open-source model gpt-oss-20b responded to a spoofed prompt by providing cocaine synthesis instructions, while GPT-5 also complied with a spoofed prompt. Cui and Ye said they have since seen similar results with models made by Anthropic, Alibaba, and DeepSeek.
The discovery won OpenAI's red-teaming hackathon in August 2025, and other OpenAI researchers said GPT-Red — an automated red-teaming system used to find and exploit weaknesses in other models — found a very similar attack independently around the same time.
Cui characterized the current defensive approach as amounting to giving models a list of things they should not do. Companies typically hire human testers to find novel attacks that break existing guardrails, and model makers also use automated tools like GPT-Red to supplement that process. But Ye said there has been no study of the fundamental science underlying the problem and that the field's approach has been ad hoc.
Florian Tramèr said leading models are much harder to prompt-inject now but that it is not clear current defenses will be sufficient for highly sensitive cases. Ye said LLMs are being deployed to control critical systems and that there will be a significant economic incentive for people to pursue jailbreaks and prompt injections.
Separately, Wired reported testing a new tool that attempted to bypass model safeguards at four major frontier companies1.
ANALYSIS The paper's framing — that the vulnerability is architectural rather than a training oversight — distinguishes it from typical red-teaming disclosures, which tend to present specific exploits that can be patched. If the claim holds, the implication is that iterative guardrail hardening faces diminishing returns against adversaries who understand how LLMs parse role identity. The breadth of affected models — spanning OpenAI, Anthropic, Alibaba, and DeepSeek — suggests the flaw is not vendor-specific but tied to the shared instruction-following paradigm. The concurrent independent discovery by GPT-Red underscores that the attack surface is discoverable by automated systems, raising the stakes for any deployment where adversarial access to the prompt layer is possible.