Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases

A recent AI study suggests that reusing smaller datasets can significantly accelerate model training. The study found that in terms of algorithm tasks, network architectures, and optimizer settings, reusing a small number of samples is more effective at saving computational resources than using a large single dataset. This phenomenon cannot be explained by existing theories; the speed improvement stems from the inter-layer growth effect caused by sampling bias, and it is even more pronounced in environments with small datasets. The study confirmed through theoretical analysis and multiple experimental interventions that in scenarios where data is scarce, the “small dataset plus reusing” strategy is not only a passive coping mechanism but also an advantageous inductive bias that can be actively utilized, especially in reasoning tasks.