Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
2026-09-07 12:00Science🔥 47.2 heat score
2sources
1days unfolding
47.2heat score
5mentions
SummaryAI generated
Recent research indicates that the performance decline in Layer Dropout in large language models (LLMs) is mainly due to improper configuration rather than limitations of the technology itself. By optimizing key factors such as hyperparameter scaling factors, block dropout granularity, layer distribution, and time scheduling, this strategy can significantly reduce validation loss with the same training floating-point operations; or save up to 25% of training computational resources while maintaining the same validation accuracy. Additionally, the optimized Layer Dropout technique effectively supports inference optimization techniques such as early dropout, skipping intermediate layers, and self-investigation decoding, achieving up to 1.5 times faster inference with negligible accuracy loss. These conclusions have been verified in over 2,400 experiments involving models with 271 million to 8.2 billion parameters and large-scale datasets, all of which were conducted on the Cerebras CS-3 system.
This study proposes that Layer dropout should be used in the training of state-of-the-art large language models (LLMs). Experiments show that, when optimizing layer distribution, time scheduling, and optimizer hyperparameters, Layer dropout can reduce loss values; for a given number of training steps, the model can achieve lower or…
The study demonstrates that the decline in performance due to layer discarding (stochastic depth) stems from improper configuration, rather than inherent limitations of this method. By optimizing the formula, layer discarding can reduce validation loss with the same training FLOPs or save up to 25% in FLOPs for the same validation loss. Experiments verified four key factors on models with 271M to 8.2B parameters and a token size of 160B: the hyperparameter scaling factor 1/(1−p), the granularity of layer discarding, increasing layer distribution (ILD), and reducing time scheduling (DTS). Larger models can tolerate higher maximum discarding rates, and the benefits increase with scale. Additionally, this technique supports cost-free推理 depth elasticity, enabling early exit and skipping intermediate layers, resulting in up to 1.5 times faster inference and negligible accuracy loss. All experiments were conducted on the Cerebras CS-3 system.