Canalization Before Generalization: Grokking as a Dynamical Probe

On September 7, 2026, arXiv published a paper titled “Canalization Before Generalization: Grokking as a Dynamical Probe”. This study focused on the “Grokking” phenomenon in over-parameterized neural networks (i.e., the separation of training fit from performance on unseen samples), and applied short-term fixed-length weight decay perturbations to three tasks to measure its impact on later generalization time. The study found that early in the pre-generalization phase, these effects manifested as a disordered state; subsequently, a stable dose-order was formed: stronger WD increased earlier generalization, while weaker WD led to later generalization. This orderliness occurred before visible generalization in all three tasks. Additionally, when the test loss barrier collapsed to zero between the perturbed and baseline generalization checkpoints, the aforementioned time effects continued to exist. The authors used Waddington’s analogy of developmental landscapes to describe this increasingly constrained choice of solution and its continuous…