Optimization for Overparametrized Models Recently, there has been a surge in interest in developing optimization algorithms for overparameterized models as achieving generalization is believed to require algorithms with suitable biases. This interest centers on minimizing sharpness of the original loss function; the Sharpness-Aware Minimization (SAM) algorithm has proven effective. However, existing literature focuses on only […]
Nonconvex Optimization and Transformer Architectures Deciding whether saddle points exist or are approximable for nonconvex-nonconcave problems is usually intractable. Zhang, Zhang & Sra [SIAM Journal on Optimization 2023] takes a step toward understanding a broad class of nonconvex-nonconcave minimax problems that do remain tractable. Specifically, it studies minimax problems in geodesic metric spaces. The first […]
Symplectic Optimization Geometric numerical integration has recently been exploited to design symplectic accelerated optimization algorithms by simulating the Bregman Lagrangian and Hamiltonian systems from the variational framework introduced by Wibisono et al. In Duruisseaux & Leok [OMS 2023] the authors discuss practical considerations that can significantly boost the computational performance of these optimization algorithms, and […]
Sampling for Constrained Distributions with Applications Mangoubi and Vishnoi [COLT 2023] considers the problem of approximating a d×d covariance matrix M with a rank-k matrix under differential privacy constraint. The authors present and analyze a complex variant of the Gaussian mechanism and give the optimal bound on the Frobenius norm of the difference between the […]
