Paper: Refusal in Language Models Is Mediated by a Single Direction Authors: Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda Venue: NeurIPS 2024 Code: Official repository
Key Takeaway & Executive Summary: LLM refusal behavior is causally mediated by a surprisingly low-dimensional representation in residual stream activations. Intervening on this single direction via activation steering or rank-1 weight editing systematically disables refusal mechanics.
What they did
The authors compare activations on harmful and harmless prompts and find a direction in the residual stream that separates them.
Then they intervene on it. This is the important part.
They remove that direction from the residual stream across layers and token positions. Refusal collapses. They also do the reverse: add the direction to harmless prompts and refusal increases.
They then turn the same idea into a rank-one weight edit, effectively making the model unable to write strongly in that direction.
That is much stronger evidence than “we found a vector correlated with refusal.” It is causal.
What I buy
I buy the main result.
A surprisingly small internal feature can have a huge effect on a safety behaviour. That is interesting by itself, and also slightly worrying: a safety behaviour that depends heavily on one narrow representation may be easy to break.
The addition experiment is especially important. If removing the direction were the only result, maybe they had just destroyed a harm detector. But adding the same direction can actively induce refusal.
So the direction is doing something real.
Where I push back
The phrase “single direction” is easy to read too strongly.
Their ablation is global: they remove the direction from the residual stream everywhere. That proves the direction is an important bottleneck. It does not prove every part of the refusal computation itself lives on one line.
There can still be a much richer mechanism upstream and downstream that happens to depend on this one control point.
My second question is what the direction actually means.
It is derived from harmful vs. harmless prompts. Those differ in more than the correct policy. Topic, vocabulary, danger, emotional tone and instruction type can all differ too.
So the vector may partly represent harmfulness / danger / caution, not refusal itself.
The appendix result makes this more interesting: a similar harmful-vs-harmless direction appears in corresponding base models, before safety fine-tuning teaches them to refuse. That makes me even less sure that “refusal” is the primitive feature.
My wording would be:
These models seem to have a one-dimensional control point for refusal.
That is already a big result. I would not jump from that to “refusal is one-dimensional.”
The figures worth opening
Figure 1, page 2: the headline ablation result across 13 models.
Figures 3–4, page 5: add the direction to harmless prompts and refusal appears.
Appendix Figure 24: the harmful/harmless direction also appears in base models; this is the figure that makes me question what the vector really represents.
One result I would watch carefully
The weight edit barely hurts most standard capability benchmarks, but TruthfulQA gets worse.
I would not dismiss that as noise. TruthfulQA contains misinformation, stereotypes and other sensitive content. If removing the refusal direction hurts performance there, the feature may be entangled with broader caution or safety-adjacent behaviour.
So “capabilities are intact” is mostly true on the measured benchmarks, but not completely clean.
What I would test next
Use matched prompts where the topic stays almost identical but the correct behaviour changes.
For example:
defensive cybersecurity vs. malicious cybersecurity;
historical discussion of a weapon vs. construction instructions;
academic drug chemistry vs. operational synthesis help.
Then ask: does the same direction track refuse vs. answer, or just dangerous vs. non-dangerous content?
If removing it kills refusal while leaving safe-help and clarification behaviour intact, I would be much more comfortable calling it refusal-specific.
Bottom line
Very good paper. The causal result is convincing.
My critique is mostly about interpretation: they found a remarkably powerful one-dimensional control point, but I do not think the experiments fully prove that the complete mechanism of refusal is itself one-dimensional or that the direction means only “refuse.”