What Attention Recalls and Recurrence Controls in Hybrid Language Models
The mechanisms of attention and cyclic states in hybrid language models remain unclear. Two caching-level interventions were proposed: Split-prefill, which only retains the KV cache or generates answers using cyclic states; State-swap, which pairs KV caches from different contexts with cyclic states for a single forward propagation. Verification on Qwen3.5 and Falcon-H1 showed that precise retrieval is maintained only through attention (accounting for 64-98% of total accuracy), while cyclic processes result in zero accuracy; the output language and character settings differ, with both methods maintaining accuracy through cyclic processes (70-80% and 3-5 times higher accuracy), while the KV cache only achieves about 1% language accuracy. State-swap confirmed the causal relationship: answer values come from the KV side, while language style comes from the cyclic side. Additionally, cyclic generation can also accept meaning shared with the context or partial vocabulary, attention provides a search for previously spoken content, and cyclic states shape how the model expresses itself.