Towards Understanding Pause Token Fine-Tuning Dynamics: A Mode Retention Perspective

The study proposes the Masked Boundary Pause (MBP) rule, which intervenes in the training process by placing pause tokens with masked loss masks at the boundaries of inference steps. This strategy was tested on 1B-8B models such as Qwen and Llama, resulting in a 6-point improvement in mathematical task performance and a 2.5-point improvement in code tasks, while maintaining general language understanding capabilities. Experiments show that MBP optimizes training dynamics through a pattern retention mechanism: in synthetic continuous learning tasks, masked pause tokens only rewrite approximately 4 times the previously learned distribution when matching the final fitness; in mathematical inference tasks, boundary-neighboring tokens encode more information about downstream steps. Additionally, this pattern retention strategy extends benefits to GRPO, indicating that pause tokens are tools for intervening in training dynamics rather than merely computational tools during inference.