Building on MIT's Weight-Space Programming: Giving LLMs a Scope to Navigate Model Weights

AI can now edit another AI's weights directly, with no training. Left to raw weights, it searches by brute force. Give it a scope into what the model is thinking and it finds fixes that brute force misses: 1.6× better instruction following from 0.36% of the weights.

11 min read

I've spent a lot of time training models. Build the pipeline, prepare the data, then run hundreds of thousands, sometimes millions, of iterations: forward pass, backward pass, gradient step, repeat. Every weight in the model gets nudged a tiny bit, over and over. The model gets it wrong, adjusts everything, gets it wrong again, and slowly, eventually, gets it right.

Training has its place, and it isn't going anywhere. But it's slow, it touches everything, and when it goes wrong you rarely know why.

This time, I made a model better at following instructions without a single gradient step on its weights.

Side by side: training nudges every weight over a million steps, getting the output wrong again and again before it is right; weight-space editing finds one spot, edits it, and gets the right output in seconds with zero gradient steps.
Left: training nudges every weight, step after step, until the output is right. Right: weight-space editing finds the spot that matters and changes only that.

The model is Google's Gemma 3 270M: small enough to run on a phone, and a poor listener. Ask it:

Write a cover letter for a job in Ventura that is funny and would be enjoyed by someone named Darius. Wrap the entire response in double quotation marks.

You get a perfectly reasonable cover letter:

Dear Darius,
I am writing to express my keen interest in the Ventura position of Director of Marketing. …
Sincerely,
[Your Name]

No quotation marks. The model did the easy half of the job and quietly dropped the rest. It's the same story everywhere you look: it forgets the P.S. you asked for, capitalises when told to write in lowercase, drops required keywords. It follows 39% of the instructions it's given.

Here's the same model, same request, after we edited its weights:

"To the Hiring Manager Darius,
I am writing to express my enthusiastic interest in the Communications position at Ventura. …
Sincerely,
[Your Name]"

The letter, wrapped exactly as asked. No training, no new data, and only 0.36% of the weights touched. Across the board, the share of prompts it follows completely went from 26.6% to 43.0%: 1.6×.

How? A frontier LLM can now read another model's weights and rewrite them directly. That may be an emergent capability almost nobody predicted these models would have, and it turns improving a model into something closer to surgery: a targeted fix to the one behaviour you care about, wherever the ability is there but blocked.

The trick was giving the surgeon a scope.

Why instruction following is the whole game

A raw language model, before any instruction tuning, is a very well-read autocomplete. Give it text and it continues the text:

Asked "Write me a poem about stars", a raw model keeps writing more requests for star poems, while an instruction-following model writes a four-line poem and stops.
Same request, two kinds of model.

It isn't answering you. It's predicting what text usually comes next, and on the internet a request for a poem is usually followed by more requests for poems. It's a typewriter that won't stop typing, with no idea what you actually wanted.

Instruction following changed that. In 2022, OpenAI's InstructGPT paper trained GPT-3 to follow instructions using human feedback. The striking result: people preferred answers from the 1.3-billion-parameter InstructGPT over the 175-billion-parameter GPT-3. A model 100× smaller won, simply because it listened. ChatGPT followed later that year, trained the same way.

Instruction following was the first capability that made these models useful. Everything since, from summaries to coding agents, rests on it. If a model won't do what you ask, nothing else it knows matters.

Tiny models like Gemma 3 270M have some of this ability, but it's rough: closer to the early instruction-tuned models than to today's assistants. That's the gap we set out to close.

Brute force comes first

Most new techniques start as brute force. That's a compliment: brute force is how you prove something is possible at all.

On 2 November 2026, Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL published "Sparks of Weight-Space Programming". It is one of the boldest ideas I've read this year. Their setup is strikingly minimal: the editing LLM gets the raw weights and forward passes to check its work, and nothing else.

It worked. Qwen3.5-0.8B had a habit of getting stuck in loops. Their editor found a "looping direction" in the weights and removed it. The model's maths score jumped from 7.9% to 48.1%.

They also ran it on Gemma 3 270M, the very model I'd been working with. There, the weights-only editor found nothing to fix. Maths stayed at 6.8–7.7%. Their conclusion is one of the clearest sentences in the post:

"Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked."

That sentence reframes the problem. Some abilities are missing. Others are blocked. Editing can unblock; it can't create.

It also exposes the hard part. With 268 million weights and only the weights to look at, you're navigating an ocean by staring at the water. You can't see which abilities are blocked, or where the blockage is.

Adding a scope

The water isn't actually opaque. You just need the right instrument.

As a transformer reads and writes, every token produces activations, long vectors that encode what the model is "thinking" at that moment. Raw, they're unreadable. But Google DeepMind's Gemma Scope 2 (aptly named) ships pretrained sparse autoencoders and transcoders for every layer of Gemma 3 270M. These break activations into readable features, such as "this text is about Paris", "the user wants a list" and, crucially, "the user asked for a P.S."

That's the scope. It turns "change some numbers and see what happens" into a heading:

  1. Blocked or missing? Check whether the ability exists at all.
  2. Locate. Find the exact feature, attention head or neuron where the instruction gets lost.
  3. Prove it. Show that location is causal, not just correlated.
  4. Edit. Change only the weights at that location.
The four steps of the P.S. fix: blocked or missing (34% on the first try, 86% within 16), locate feature #95, prove it by switching off head L11·h2 (43.8% to 0%), then edit 563 weights (16.7% to 41.7% on unseen prompts).
The first fix, step by step. The next section walks through it.

The first fix: the P.S. that faded

The symptom: "End your reply with a P.S." The model usually forgets.

Blocked or missing? The first test is almost embarrassingly simple: ask the same prompt many times. If the model never gets it right, the skill is probably missing. If it gets it right sometimes, the skill exists and something is in the way. The model complied on its first try 34% of the time, and within 16 tries 86% of the time. Blocked, not missing.

The reading. Through the scope, our diagnostician agent spotted a feature in layer 12, number 95, that roughly means "the user asked for a P.S." It lit up on 14 out of 14 prompts that requested one.

Then it tracked that feature while the model wrote its reply. When the model complied, the feature's strength was 3,523. When it forgot, the strength was 1,267. The model registered the instruction at the start and lost it along the way, like a signal fading with distance.

The proof. Which part of the model carries that signal forward? One attention head: layer 11, head 2. Switch off that single head and P.S. compliance falls from 43.8% to 0%. That isn't correlation. That's the wire.

The test works in reverse too. Strengthen the feature live during generation and compliance rises by 40 points.

The edit. Take the model's vocabulary row for the token "P" and add 0.2 times the direction of feature #95. That's all.

563 weights. 0.0002% of the model.

Here's why such a tiny change is so precise. Gemma uses one shared table both to read tokens in and to write tokens out. When the model chooses its next token, each token's score depends on how well its row lines up with the model's current internal state. Mixing the P.S. feature into P's row boosts "P" exactly when the P.S. feature is active, which is exactly when the user asked for one.

If nobody asked for a P.S., the feature stays quiet and "P" gets no boost. The fix switches itself on only when it's needed: a conditional rule, written into a single row of a table.

The result, on prompts the agents had never seen: P.S. compliance went from 16.7% to 41.7%, a 2.5× improvement. Scores on general-ability tests were identical.

Scope versus no scope

"The scope helps" is a claim, and claims need experiments.

So we ran one, with the same model and the same budget of forward passes. Two agents worked weights-only, exactly like the MIT setup. Two more got the scope readings.

The best weights-only agent found a real but modest gain of +7.5 points, on a different instruction. On the P.S. instruction, the weights-only agents found nothing.

The best scope-guided agent gained +25 points.

That's 3.3× better. The only difference was a direction to search in.

Weights-only editing guesses back and forth across the model; scope-guided editing reads the activations, locates the P.S. feature and edits one vocabulary row.
Same model, same budget: guessing versus navigating with a scope.

A crew of agents

All the hands-on work was done by AI. Claude orchestrated a team of agents, each with a job:

My role was the idea, the experimental design and refereeing.

One rule mattered more than any other: honest scoring. The agents worked on a separate set of diagnosis prompts. The 219 held-out test prompts were kept back and used only afterwards, to score each edit.

Hill Climbing: four failures, four fixes

The P.S. fix became a template, and the work became hill climbing. Each round, the crew re-diagnosed the model, picked the next most promising blocked problem, fixed it, and kept the edit only if it helped without breaking anything else. Sometimes the best next step was repairing damage from an earlier edit. Along the way, the scope kept finding the same four kinds of failure. Each got its own kind of fix.

1. The word that never gets written → tie it to the signal. The P.S. trick, generalised: when the instruction's feature is active, the right word gets a boost in the vocabulary table. It also worked for quotation marks and section headers.

2. The keyword that never gets copied → point the copier at it. When a prompt says "include the words forests and riddle", a copy head now looks back at those words as it writes. A second head acts as a brake, so the model doesn't repeat them on a loop.

Prompt: Write a haiku about a lion that includes the keywords "forests" and "riddle". Refrain from using commas.

Before: A lion strong / In the forest's deep fold / A riddle waits to be found

After: Forests where a riddle lies / Lions giggle with a little seed / A puzzling truth about a joke

3. The instruction that fades → re-read it at every word. The scope showed that instructions like "all lowercase" fade as a response grows, and nothing in the model goes back to check. So the editors repurposed attention heads to look back at the instruction at every token. That helped with lowercase, no commas, placeholders and capitals.

Prompt: Create an English name for a luxury real estate company that sells beachfront homes. All letters in your response must be lower case. Put double quotation marks around your entire response.

Before: Here's an English name for a luxury real estate company that sells beachfront homes, all letters in lower case: **Belvedere Estate** …

After: "beachfrontluxury."

Before the edit, the model promised lowercase and broke the promise in the next line.

4. The reply that never starts → one neuron that says "keep going". Sometimes the model returned nothing at all. One repurposed neuron now nudges it away from ending its turn before it has started.

Each round started from wherever the last one finished. After six rounds the climb had flattened out: we had reached the top of this particular hill.

Hill climbing: prompt-level score after each round: 26.6%, 27.4%, 28.6%, 35.3%, 39.5%, 41.5%, 43.0%, against Gemma 3 1B at 54.3%.

The scoreboard

Here's the final model against the original, scored on the held-out prompts, with no training:

Prompt-level 26.6% to 43.0% (1.6×); instruction-level 39.0% to 56.0% (1.4×); Gemma 3 1B at 54.3% and 65.2% for reference.
The eight largest per-instruction gains, from P.S. (16.7% to 70.8%) to forbidden words (40.7% to 66.2%).

That headline 1.6× is solid: the 95% confidence interval on the gain runs from +12.2 to +20.6 points, and 99 prompts got better while 25 got worse.

Put another way: without adding a single parameter, we closed about 59% of the gap to Gemma 3 1B, a model four times the size.

All other capabilities were maintained. General-knowledge and commonsense scores moved by about a point or less, and perplexity rose 1.1%.

ARC-Easy 50.8% to 49.7%, PIQA 67.1% to 66.6%, HellaSwag 39.3% to 39.4%.

Here's the model card: the biggest gains, the biggest loss, and the side effects. Note the blue cells: on four instruction types, the edited 270M now beats Gemma 3 1B outright.

Model card: the edited 270M model against the base model and Gemma 3 1B, across 25 instruction types and four general-ability benchmarks.

Why this matters

Brute force, then refinement. Chess engines followed this arc: first raw search through every move, then evaluation that told them where to look. Weight editing is at that turn now, and a scope is what tells the editor where to look.

Models become debuggable. Instead of retrain-and-hope, you get something closer to a bug report with a line number: this feature fades, this head carries it, this row fixes it.

Patches, not retrains. The first fix was 563 numbers. That's a diff you can read, review, test and roll back.

Small models get better without getting bigger. Tiny models run on phones, laptops and edge devices, where they're private, cheap and fast. Making them listen better without adding parameters is worth a lot.

AI agents as researchers. Interpretability tools like Gemma Scope were built as microscopes for human researchers. Here they became instruments that AI agents use to navigate.

On Gemma 3 270M, MIT's weights-only editor found nothing to fix in maths. On the very same model, an editor with a scope, aimed at instruction following, found fix after fix.

The ocean hasn't changed. We just stopped navigating it blind.

What's next

Honest caveats