Prompt-Aware Image Token Pruning for Multimodal LLMs
Trained a 1.3M-parameter scorer that distills where LLaVA-1.5-7B attends (its layer-8 attention), pruning the image tokens the model would ignore before they reach the frozen LLM. Keeping only a quarter of the 576 image patches, it nearly matches the unpruned model on TextVQA (36.0 vs. 36.7) and beats random pruning by 12.3 points, and its forward pass takes 0.43 ms per image, against 65 ms for a CLIP-GradCAM scorer. An alignment diagnostic across three VQA benchmarks showed why the training-free shortcut fails: what CLIP finds salient barely tracks what the LLM attends to. A Rice COMP 646 project with Chao Hsuan Ho and Kerstin Sun.
~10× less LLM attention compute (estimated) · 0.43 ms scorer pass per image · +12.3 pts over random pruning