![]()
ArXiv
Preprint
![]()
Source Code
Github
A vision-language model, or VLM, looks at a picture and answers questions about it in words. Ask a VLM what is in a photo, and it names things one at a time: a dog, then a cat, then a horse. It almost never names the same thing twice, and it rarely skips one. But a picture has no reading order. Nothing in a photo says the dog comes before the cat. So how does a VLM decide what to describe first, what comes next, and when it is done?
We opened up a VLM and traced how it does this. We found that the VLM decides where to look before it looks. A few small teams of parts inside the VLM work together. One team finds the right places in the picture. Another carries those places to the moment the VLM is about to answer. A third looks back at the picture and reads what is there. And after every answer, the VLM keeps a simple bookmark of how far it has come.
You give a VLM a picture and a question, and it writes an answer in words. Inside, it chops the picture and the question into small pieces called tokens and passes them up a stack of 36 layers. Small parts called attention heads let each token pull in information from the tokens before it. That is all you need to follow along. If you want to see how it works, the primers below zoom in one level at a time: the transformer block, the attention inside it, one attention head, and how a picture becomes tokens.
Almost every modern language model, VLMs included, is a transformer. It reads text as a row of tokens, small word pieces. Each token carries a running note of everything the VLM knows about it so far. This note is called the residual stream, and you can picture it as a straight line running up through the VLM.
This is the simplest transformer block. It does two things, in order. Attention reads the stream, pulls in information from other tokens, and adds it back. Then the MLP reads the stream and works on this token alone, and adds its result back too. A model is just many of these blocks stacked on top of each other. Ours has 36, which we call its layers.
Here is a whole sentence going through two blocks. We follow the last token.
Zoom in one level: open the attention box from the transformer block.
Zoom in one more level: open a single attention head. Each head does a small search over the earlier tokens, using three things: a query, keys, and values. Click through to follow one head at the word it, step by step.
Everything so far was about words. Here is how a picture gets in.
A VLM writes its answer one token at a time, and every new token goes through the whole stack again. This creates a real puzzle. The VLM has no notepad. It can't write down "I already said dog" and look at it later. Every time it starts a new word, it has to work out again, from the words it has already written, what is left to say.
So how does a VLM pick the next thing to describe, and how does it keep track of what it has already described?
Here is the answer in one picture. Click through it one step at a time. In the sections after it, we explain each step and how we found it.
Photos are messy. We can't control which objects the VLM will mention, or in what order, so it is hard to test anything cleanly. Instead, we built a simple puzzle where we control everything. Each picture is a strip of panels. Each panel holds one shape, with a letter written inside it. We ask the VLM for the letter inside a named shape.
Notice that we ask about a shape, not about "panel 2". To answer, the VLM first has to find where the star is, and only then read the letter. That gives us two separate jobs to look for inside the VLM: finding the place, and reading it.
The puzzle also has a list version. When the same shape shows up in several panels, we ask for the letter inside each heart, and the VLM lists them from left to right. This is where it has to remember what it has already written.
Later, we check that everything we find on these strips also happens on real photos.
We can't ask the VLM what it is thinking. But we can do something better. We can copy a token's state from one run of the VLM and paste it into another run, at the same token and the same layer, and see how the answer changes. This is called activation patching.
Concretely, we run the VLM on two puzzles. The source run asks for the heart in one picture, and the target run asks for the star in a different picture. We paste one state from the source into the target and read the target's new answer. We design the two pictures so that each kind of information the state could carry makes the target say a different letter. So the letter tells us what was inside the state.
Activation patching is how we test what a piece of the VLM's work contains. Click through to see one patch.
Patching only works if the answer tells us what the pasted activation carried. So we build every pair of puzzles so that each kind of information it could carry leads the target to a different letter.
The source's heart sits in panel 4, and the target's star sits in panel 2. Every letter in the target means something different:
One more check runs through every experiment. Whenever we paste a team of heads, we also paste the same number of heads picked at random from the same layers. If the random heads do nothing, the effect belongs to the team we found, not to pasting heads in general.
Our first patch is at the colon after "Answer", the last token before the answer, one layer at a time. Low in the stack, pasting changes nothing. In the middle, around layers 23 to 29, the target answers K: the pasted state says "the answer is in panel 4", and the target goes and reads its own panel 4. Near the top, the target simply copies the source's letter D. So halfway through, the VLM already knows where the answer is, but has not yet read what it is. We call that kind of "look here" note a pointer.
If the colon already holds a pointer by the middle layers, the pointer must have been made somewhere before it. So we repeated the pasting at every token of the question, at every layer. Only one other token lit up: the shape word itself, star. The VLM works out where the star is while it is still reading the question.
Next, we zoomed in on the attention heads at that word. Concretely, we pasted the output of just one attention head at a time, for every one of the VLM's 1,152 heads, and counted how often the target switched to K. A small team of 16 heads, in layers 15 to 21, did almost all the work. Pasting just those 16 moves the pointer most of the time. As a check, we also pasted 16 heads picked at random from the same layers, and they did almost nothing. We call the 16 heads binding heads, because they bind a word in the question to a place in the picture.
We could even split their job in two. Remember that a head matches its query, what it is looking for, against the keys of the image tokens, the labels they carry. When we pasted only the queries, the target searched for a heart in its own picture and answered P. When we pasted only the image keys, the target searched for its own star among the source's shapes and answered R. When we pasted both, the pointer itself moved. In plain words, the head says "I need a star", and the patches answer "I am the panel with the star."

Here is a gap. The pointer is made at the word star, but the answer is written several tokens later, at the colon. Remember that attention heads are the only way information moves between tokens. So some heads at the colon must reach back to star and fetch the pointer.
Concretely, we let one head at a time at the colon read from the source run instead of the target, and counted how often the target switched to K. Another team of 16 heads stood out. When we let these heads read from only one token at a time, they got the pointer mostly from star, exactly where the binding heads left it. We call them relay heads.
Two more tests tie the teams together. If we stop the relay heads from looking at the question, the VLM almost always loses its answer. And here is the cleaner one: we move the pointer with the binding heads in a separate run, and then let only the relay heads of a fresh target read from that run. Nothing else gets through. The target still follows the moved pointer just as often. So the relay heads alone carry the pointer from the question to the answer.
At the colon, the VLM now holds a pointer, but the letter is still sitting in the picture. Something has to follow the pointer back and read it. Concretely, we pasted only the query of one head at a time at the colon, and found 32 heads, all in the upper half of the VLM, that send the target to K. We call them gaze heads, because they decide where the VLM's gaze lands.
They behave exactly like someone following directions. If we move the pointer with the binding heads but hold the gaze heads fixed, the moved pointer does nothing at all: it only works through them. And if we let the gaze heads look at the source's picture while following the target's pointer, the target reads out the source's letter. Many of these heads are the same ones we found in earlier work, Gaze Heads, by watching where a model looks while it narrates a comic.
When the gaze heads read a panel, do they bring back the word for what they saw, or something more like the idea of it? To test this, we dropped the letters and asked about colored shapes instead, "What shape is the green marker?", in English, German, French, and Chinese. The VLM always answers in the language of the question.
The gaze heads don't care about that language. What they bring back looks almost the same for one picture asked in two languages, and very different for two pictures asked in one language. If we paste the gaze heads from a source that asks in French about another picture, the target names the source's shape, but in its own language. So the gaze heads bring back the idea, not the word.
The language comes from somewhere else. A separate team of 16 language heads, near the top of the VLM, carries the language of the question. Paste them, and the answer switches language while the shape stays the same. The last few layers then put the idea and the language together into the final word.

So far the VLM gives one answer. Now ask for the letter in each heart. After it writes E, it has to know that the first heart is done and move on. And since it has no notepad, it has to work this out again from the letters it has already written.
The only tokens that know what has been written are the letters themselves and the commas after them. So we pasted the comma from a source that had written one letter into a target that had written nothing. Around layers 20 to 21, the target skips a panel it never answered from. The comma carries the VLM's progress.
But what does that progress look like? It could be a count, "two done". It could be a list, "panels 2 and 4 done". Or it could be a place, "done up to panel 4". We built pairs of pictures where each of these would make the VLM say a different letter. The answer was clear: it is a place. The comma holds a bookmark that says "done up to here", and the VLM moves on to the next matching panel after it.
Who sets the bookmark? The same binding heads. At each written letter, they link the letter back to the panel it came from. If we quietly move a written letter one panel to the right in the source picture, the target's next answer also jumps one panel to the right. The binding heads do the same job twice: first they tie the question to the picture, then they tie each answer to the spot it came from.

Last comes the choice of what to read next. We can watch it without pasting anything, just by handing the VLM letters out of order. With only the second heart written, it goes on to the third. With only the last heart written, it goes back for the earliest one it skipped. The rule is simple: take the nearest unread panel after the bookmark, and go back only when nothing is left ahead. When nothing is left at all, the same step tells the VLM to stop.
A strip of panels is a clean lab, but people ask about real photos. So we asked the VLM ordinary questions about everyday photos, like "What animals are in this image?". The gaze heads look ahead at the animal the VLM is about to name. If we stop the comma from seeing an animal that was already named, the VLM names it again. Setting the language heads to English turns answers in other languages into English. And the binding, relay, and gaze teams show up in every size of this model we tried, from 2 billion to 32 billion parameters.
Most of our careful tests use the panel strips, where we control every detail. On photos we can show that the same teams matter, but we can't build pairs that are as clean. We also didn't find a small team that makes the final choice of the next panel. That choice is spread over several layers. We can find it and change it through the comma, but not pin it to a few heads the way we can for the other steps. On the largest model, the binding work is spread over more heads. And the language heads hold up on the middle-sized models, but not on every size.
For all the experiments, checks, and numbers, see our paper.
Earlier work found pieces of this picture in text-only models. Language models keep track of which thing goes with which detail by tagging each thing with a hidden label and following that label later (Feng and Steinhardt, 2024). They use a similar trick to follow who believes what in a story (Prakash et al., 2026). And they handle lists with filter heads (Sen Sharma et al., 2026), heads that pick out the items of a list that match a rule. Our binding heads do something similar over the panels of a picture.
In models that see pictures, Assouel et al. (2026) found that the words of a question build an index of matching places in the picture. Our own earlier work on Gaze Heads found heads that look at the part of a picture being described, so that turning them turns the description.
These works explain how a model finds one place that answers a question. Here we show how these parts work together when the answer is a list, and we add the missing piece: how the VLM keeps its place in the picture and chooses what to read next.
The paper can be cited as follows.
Rohit Gandikota, Arnab Sen Sharma, Adam Belfki, David Bau. "How VLMs Plan What's Next." arXiv preprint, 2026.
@article{gandikota2026plan,
title={How VLMs Plan What's Next},
author={Gandikota, Rohit and Sen Sharma, Arnab and Belfki, Adam and Bau, David},
journal={arXiv preprint},
year={2026}
}