Generate
The compressed student produces a response using reduced visual inputs.
arXiv 2026 / Visual token efficiency
Fewer Tokens,
More Self-Teaching.
On-Policy Self-Distillation for
Extreme Visual Token Reduction
Shanghai Jiao Tong University · University of Cambridge
Xidian University · Xi’an Jiaotong University
5%
visual tokens retained
82.3%
full-model performance retained
85.4%
less prefill FLOPs
Paper results · Qwen3.5-4B · averaged over nine benchmarks
RELEASE NOTE
October 5, 2026 — New checkpoint with learnable token merging: 84.5% retained performance, as reported in the repository. Separate from the paper’s 82.3% result.
Explore release ↗01 / THE IDEA
Teach the model
where it actually goes.
Extreme compression changes both what a model sees and the responses it produces.
LT-OPD trains a low-token student on its own generated prefixes, supervised by a frozen full-token copy. A decreasing token-budget curriculum supports adaptation. At inference, the compressed student runs alone.
Read the full abstract and paper ↗02 / METHOD
Same model. Different visual budgets.
Low-Token On-Policy Distillation
Self-teach
A frozen teacher supplies next-token distributions on those same prefixes.
Adapt
Token-level Jensen–Shannon divergence aligns the student with its teacher.
A curriculum for compression
175 updates · 25% → 5% visual tokens
25%Warm start / 14 updates
25% ↘ 5%Cosine decay / to update 100
5%Target budget / final 75 updates
03 / RESULTS
Recover capability.
Keep the efficiency.
Qwen3.5-4B · 5% visual-token budget
Average retained performance
100% denotes the full-token Qwen3.5-4B reference.
See all nine benchmark scores +
| Method | V*Bench | HR-4K | GQA | MMMU | MMBench | MME | POPE | TextVQA | OCRBench |
|---|---|---|---|---|---|---|---|---|---|
| Full tokens | 84.3 | 84.4 | 60.7 | 77.6 | 86.4 | 2300.8 | 89.3 | 81.2 | 850 |
| CDPruner · 5% | 52.9 | 68.4 | 50.4 | 45.7 | 74.0 | 1531.8 | 70.2 | 43.2 | 404 |
| LT-OPD · 5% | 74.9 | 73.0 | 54.8 | 50.0 | 78.9 | 2077.0 | 86.3 | 56.0 | 535 |
Retained performance = mean of each benchmark score divided by its full-token reference score. It is not an overall accuracy. Source: paper, Tables 2–5 ↗
Evaluated across model scales and familiesQwen3.5-9BGLM-4.6V-9BLLaVA-OV-1.5-4B
04 / OPEN RESOURCES
From paper to practice.
Code, weights, and training data
01 / GITHUB ↗
LT-OPD code
Training, compression runtime, checkpoint export, and benchmark evaluation.
Apache-2.002 / HUGGING FACE ↗Model checkpoint
Current Qwen3.5-4B release with CDPruner and learnable token merging.
Use the project’s compression runtime03 / HUGGING FACE ↗LT-OPD-14K
14,000 training examples with packaged image shards, provenance, and checksums.
Browse the dataset and release filesGet started
Python 3.12 + CUDA. Follow the model card for environment setup and compressed inference.
Full model quick start ↗Evaluation guide ↗
git clone https://github.com/Yrxxxxxxxx1007/LT-OPD.git cd LT-OPD # Download the current checkpoint hf download yyy051007/LT-OPD --local-dir models/LT-OPD # Prepare the training data after installation lt-opd data --dataset yyy051007/LT-OPD-14K \ --output data/LT-OPD-14K
05 / CITATION
BibTeX
@article{li2026fewer,
title={Fewer Tokens, More Self-Teaching: On-Policy Self-Distillation
for Extreme Visual Token Reduction},
author={Li, Junxian and Yang, Ruixuan and Zhang, Tianao and
Xu, Tiange and Dong, Weisheng and Zhang, Yulun},
journal={arXiv preprint arXiv:2609.32353},
year={2026}
}