Optimization for ML and AI Seminar: A non-equilibrium phase transition with broken ergodicity leads to double descent and benign overfitting in machine learning
Nigel Goldenfeld, UC San Diego Department of Physics and HDSI
Abstract: The remarkable ability of modern neural networks to generalize improves with increasing network capacity, even when the number of model parameters or effective degrees of freedom exceeds the number of training data points. This phenomenon is all the more surprising given that generalization error diverges when the number of model parameters approaches a critical value from below. Here we use dynamical mean field theory to show, in a simple setting of linear regression, that this so-called “double descent” behavior is the outcome of a phase transition in the stochastic field theory describing the training process. We calculate the critical exponents and scaling function of the double descent phase transition, and show that it is marked by a breakdown of the fluctuation-dissipation theorem associated with broken ergodicity. The corresponding response function has the same functional form as the simple London model of the superconducting transition, with the rigidity of the wave function corresponding to the neural network’s ability to generalize accurately. Our results are distinct from earlier work, because we calculate the time-dependence specifically, not just the equilibrium solutions. This is what enables us to identify the origin of the emergent behavior.
Work performed with Chan Li.
Nigel Goldenfeld holds the Chancellor’s Distinguished Professorship in Physics at UC San Diego, which he joined in Fall 2021 after 36 years at the University of Illinois at Urbana-Champaign (UIUC). Nigel’s research spans condensed matter theory, the theory of living systems, hydrodynamics and non-equilibrium statistical physics.
Nigel received his PhD in theoretical physics from the University of Cambridge (UK) in 1982, and from 1982-1985 was a postdoctoral fellow at the Institute for Theoretical Physics at UC Santa Barbara, where his work on the dynamics of snowflake growth helped launch the modern theory of pattern formation in nature. He joined the condensed matter theory group at the Department of Physics at UIUC in 1985, where his work was instrumental to the discovery of d-wave pairing in high temperature superconductors. Nigel’s interests in biology include microbial ecology, evolution and systems biology. He was a founding member of the Institute for Genomic Biology at UIUC, where he led the Biocomplexity Group and directed the NASA Astrobiology Institute for Universal Biology. During the COVID-19 pandemic, he pivoted from his experience in mathematical modeling of bacteria and viruses to computational epidemiology, advising the Governor of Illinois, and helping devise, set up and run the COVID saliva testing system at UIUC, which provided ~12 hour turnaround of PCR tests to the 50,000 people in the campus community and eventually to over 1700 schools and other institutions in Illinois and beyond. Nigel has served on the editorial boards of several journals, including The Philosophical Transactions of the Royal Society, Physical Biology and the International Journal of Theoretical and Applied Finance. Selected honors include: Alfred P. Sloan Foundation Fellow, University Scholar of the University of Illinois, the Xerox Award for research, the A. Nordsieck award for excellence in graduate teaching and the American Physical Society’s Leo P. Kadanoff Prize 2020. Nigel is a Fellow of the American Physical Society, a Fellow of the American Academy of Arts and Sciences, a Fellow of the Royal Society (UK) and a Member of the US National Academy of Sciences.
Zoom: https://bit.ly/Opt-AI-ML