A single video can become tens of thousands of visual tokens before a vision-language model has answered a single question about it. Our new paper, CRAFT, is about cutting that count by roughly 8× without deleting the details that temporal reasoning depends on.

CRAFT Research · 4 min read · Read the paper on arXiv

Video VLMs have a token problem

To understand video, a VLM has to ingest visual tokens at a scale that text never demands. The cost lands almost entirely on the prefill stage — the model must process every token before it emits anything — and on the KV cache, which then has to hold all of them for the rest of the conversation. Both scale with sequence length, and video makes that sequence enormous.

The obvious response is that video tokens are highly redundant along the spatio-temporal dimension, so many of them can go. That is true, and it is also where the difficulty starts. Push the compression ratio up and you begin removing the details that distinguish one moment from another — exactly the information temporal reasoning is made of. Efficiency and fidelity pull against each other.

Existing methods resolve that tension by picking a side. Heuristic, training-free compression is cheap to deploy but adapts poorly to what is actually in the video. Methods that add learnable modules adapt better, but require expensive alignment training to make their outputs legible to the pre-trained language model. Neither settles the trade-off; each just chooses which half to sacrifice.

Separate which from how

CRAFT's central move is to notice that token merging is two decisions, not one, and that they do not need the same machinery.

Which tokens should merge? Global similarity decides. This part stays parameter-free — no training, no learned selection policy. Redundancy is a property of the tokens themselves, and measuring it directly works.

How should they merge? This is where the learning goes. A position-aware weighting module and a content-adaptive channel-wise gate decide how the selected tokens combine — which is to say, what gets preserved when several tokens become one. The gate operates per channel, so the model can hold on to the feature dimensions that matter for a given piece of content instead of averaging everything down uniformly.

Splitting the decision this way is what lets CRAFT be adaptive without the alignment-training bill. The expensive, content-sensitive judgment is concentrated in the fusion step, where it earns its keep; the selection step stays free.

Why fusing beats dropping

The distinction between pruning and fusion turns out to matter more than it first appears. Because every token CRAFT retains is a linear combination of the original tokens, two properties come along for free.

The retained tokens keep their true spatio-temporal coordinates — a merged token still knows where and when it came from, so positional structure survives compression rather than being approximated afterward. And they stay inside the input distribution the language model was pre-trained on, which is the practical reason CRAFT does not need a realignment phase: it never hands the LLM anything that looks unfamiliar.

A pruning method, by contrast, makes an irreversible bet. Whatever it discards is simply gone, and if a later question happened to depend on it, there is no recovering it.

Compress once, answer many times

CRAFT's pipeline is recursive and, importantly, query-agnostic: compression does not look at the question being asked. That sounds like a limitation and is in fact the deployment story. A query-conditioned compressor has to redo its work for every new question, which means the cost you were trying to avoid returns on each turn.

Compress a video once with CRAFT and the result is reusable across every question that follows. For anything conversational — a user asking five things about the same footage — the saving compounds rather than resetting.

What the numbers show

Across multiple representative video benchmarks, CRAFT consistently outperforms prior state-of-the-art token-compression methods. The headline operating point: at roughly 8× compression, CRAFT retains 96.8% of the backbone's average accuracy, with substantial reductions in both prefill cost and KV-cache size. (The paper's abstract rounds this to "roughly 97%"; 96.8% is the precise average.)

The number worth sitting with is the shape of that trade, not its size. Removing seven of every eight tokens and giving up around three points of average accuracy is a very different bargain from the one a naive pruner offers at the same ratio — and it is available without an alignment-training stage.

The broader takeaway generalizes past video: don't just drop redundant tokens — learn how to fuse them without losing the details. Redundancy is not the same thing as worthlessness, and treating the two as equivalent is what makes aggressive compression brittle.

Read the paper

CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models — Yu Chen, Xiaohong Li, Xiaole Wang, Jianjin Zhang, Jun Sun, Yafeng Deng. Available on arXiv.

Accuracy retention is reported as the average across the evaluated video benchmarks relative to the uncompressed backbone, measured at approximately 8× token compression.