Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation

To address the difficulties in reusing KV caches in long-scenario applications and the low efficiency of existing fine-tuning methods, researchers proposed a new approach using automatic segmentation and block distillation to achieve block attention generalization. The team constructed the SemanticSeg dataset, which contains over 30,000 instances. A lightweight segmentor was trained to divide input text into blocks that conform to human intuition and have controllable granularity. Additionally, a block distillation framework based on freezing the full-attention teacher model was proposed, integrating three components: block fusion tokens, block Dropout, and token-level loss weighting to mitigate the loss of boundary information and focus on sensitive Tokens. Experiments showed that this segmentor outperformed heuristic and statistical baselines, while block distillation achieved performance close to full attention while maintaining block attention, providing a scalable path for the deployment of related technologies.