Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

The researchers proposed a method to halve the number of activation experts in a fine-grained MoE model without training. This method reduces the number of experts during inference from 8 to 4 by introducing an parameter-less integer $k_2$. Testing on the Qwen3.6-35B-A3B model showed that this operation reduced the MMLU score by only 0.35 points while halving the computational burden of routing experts; completely removing normalization would lead to catastrophic consequences. This result was successfully replicated on the Qwen3.5-397B-A17B model, which has been scaled up 11 times. Analysis indicated that maintaining an appropriate reference quality is crucial, and different $k_2$ values are preferred for perplexity and downstream accuracy. The importance of expert identity far exceeds weights, and balance and domain-specific routing limit the space for expert pruning.