How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing

The `DeepSeek-V4-Flash model exhibits a specific pattern of residual flow usage in the four-stream residual path. Attention or FFN sites typically utilize only about two streams effectively, and the read/write routing dynamically changes with network depth. Research found that residual mixing primarily occurs in early layers, while layers 22 to 42 mainly independently forward each residual stream, without achieving deep mixing. Experimental interventions showed that replacing the late-stage mixer with an identity map had little impact on model performance (a 1.9% increase in perplexity), whereas replacing the early-stage mixer led to a significant performance decline (a 41% increase in perplexity). Additionally, fixing the early-stage mixer at the diagnostic mean or retaining the three highest routing weights for each site was sufficient to maintain stable model performance. This indicates that the model only achieves partial flexibility of four-stream mHC; individual blocks rarely require all four streams, and late-stage residual mixing does not provide significant benefits.