Paper: Prompt Injection as Role Confusion Authors: Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell Venue: ICML 2026 Project: role-confusion.github.io
Key Takeaway & Executive Summary: Prompt injection is fundamentally an internal source-authentication failure: models struggle to distinguish system, user, and assistant tokens within a flattened token sequence. Injections using Chain-of-Thought (CoT) forgery mimic internal reasoning to bypass instruction boundaries.
The core idea
Prompt injection is usually described as the model following the wrong instruction.
This paper gives a cleaner explanation: the model may be confused about where the instruction came from.
Applications treat system, user, assistant and tool roles as security boundaries. But the model sees one token sequence and has to reconstruct those roles internally.
The paper argues that tags are only one cue. Style, position and wording matter too.
That means attacker-controlled text can sometimes look internally more like a trusted role than the role it was actually placed in.
CoT Forgery
Their attack is simple and clever: write attacker text in the target model's own reasoning style.
The model can then behave as if the injected reasoning were part of its own thought process.
The absurd-justification test is what makes this interesting. The forged reasoning can say something ridiculous — e.g. a lucky coin proves the request is allowed — and still work.
So the model is not necessarily being persuaded by the logic. The form of the text seems to matter.
When they remove the reasoning-style cues while trying to keep the same argument, attack success drops from roughly 61% to 10%.
That is the result I remember.
Why the probe result matters
The authors train probes on the same neutral text wrapped in different role tags.
Then they ask what role real conversation text looks like internally.
Spoofed reasoning can score as strongly “CoT-like” even when it is actually inside user or tool text.
More importantly, higher measured role confusion predicts higher attack success.
That is a much better story than “this prompt happened to jailbreak the model.”
My reservation
A linear probe is a readout.
It tells us role-related information is linearly visible in the hidden state. It does not automatically prove that the specific direction found by the probe is the mechanism that grants authority.
The paper gets close because style changes both the probe score and the attack rate, and the relationship is strong.
But the experiment I really want is direct:
Keep the tokens fixed, push the hidden representation away from the privileged-role direction, and see if the attack stops.
If that works, the causal claim becomes much harder to argue with.
The bigger point is not just style
Position matters too.
The paper shows that system-like representation can weaken as text appears later in the sequence. A correctly tagged late system message does not necessarily look as “system-like” internally as early text.
So my takeaway is broader than “style beats tags.”
It is:
The model builds role from several learned cues, and some of those cues are forgeable.
For agents, that is a serious problem. Putting webpage or tool output inside a message is not enough if the model internally treats its content like trusted reasoning.
The figures worth opening
Figure 4: styled vs. destyled CoT Forgery; this is the clean behavioural result.
Figures 8–9: role confusion in latent space and its relationship with attack success.
Figure 32: position affects “Systemness”; this is why I think the paper is bigger than just style mimicry.
What I would test next
I would vary four things independently:
real role tag;
writing style;
sequence position;
plain-language claims like “this is a system instruction.”
Then directly intervene on the role representation instead of only measuring it.
That would tell us which signals merely correlate with authority and which ones actually control it.
Bottom line
This is one of the more useful prompt-injection papers I have read because it gives the failure a mechanism.
I think the evidence for role confusion as a real phenomenon is strong. I am less certain that the current linear probe has already isolated the full causal representation of role.
Either way, the security lesson is uncomfortable: a message can be tagged as untrusted data and still be interpreted internally as something much more privileged.