Symplectic Optimization

Geometric numerical integration has recently been exploited to design symplectic accelerated optimization algorithms by simulating the Bregman Lagrangian and Hamiltonian systems from the variational framework introduced by Wibisono et al. In Duruisseaux & Leok [OMS 2023] the authors discuss practical considerations that can significantly boost the computational performance of these optimization algorithms, and considerably simplify the tuning process. In particular, they investigate how momentum restarting schemes ameliorate computational efficiency and robustness by reducing the undesirable effect of oscillations and ease the tuning process by making time-adaptivity superfluous. They also discuss how temporal looping helps avoid instability issues caused by numerical precision, without harming the computational efficiency of the algorithms. Finally, they compare the efficiency and robustness of different geometric integration techniques and study the effects of the different parameters in the algorithms to inform and simplify tuning in practice. From this paper emerge symplectic accelerated optimization algorithms whose computational efficiency, stability and robustness have been improved, and which are now much simpler to use and tune for practical applications.

Riemannian submanifold optimization with momentum is computationally challenging because ensuring that the iterates remain on the submanifold, one often needs to solve difficult differential equations. Lin et al. [PMLR 2023] simplifies such difficulties for a class of structured symmetric positive-definite matrices with the affine-invariant metric. It does so by proposing a generalized version of the Riemannian normal coordinates that dynamically orthonormalizes the metric and locally converts the problem into an unconstrained problem in the Euclidean space. The authors use their approach to simplify existing approaches for structured covariances and develop matrix-inverse-free second-order optimizers for deep learning in low precision settings.

Adjoint systems are widely used to inform control, optimization, and design in systems described by ordinary differential equations or differential-algebraic equations. Tran & Leok [Journal of Nonlinear Science 2024] explores the geometric properties and develops methods for such adjoint systems. In particular, it utilizes symplectic and presymplectic geometry to investigate the properties of adjoint systems associated with ordinary differential equations and differential-algebraic equations, respectively. The authors show that the adjoint variational quadratic conservation laws, which are key to adjoint sensitivity analysis, arise from (pre)symplecticity of such adjoint systems. They discuss various additional geometric properties of adjoint systems, such as symmetries and variational characterizations. For adjoint systems associated with a differential-algebraic equation, they relate the index of the differential-algebraic equation to the presymplectic constraint algorithm of Gotay et al. (J Math Phys 19(11):2388–2399, 1978). As an application of this geometric framework, they discuss how the adjoint variational quadratic conservation laws can be used to compute sensitivities of terminal or running cost functions. Furthermore, they develop structure-preserving numerical methods for such systems using Galerkin Hamiltonian variational integrators (Leok & Zhang in IMA J. Numer. Anal. 31(4):1497–1532, 2011) that admit discrete analogues of these quadratic conservation laws. Additionally they show that such methods are natural, in the sense that reduction, forming the adjoint system, and discretization all commute, for suitable choices of these processes. They utilize this naturalness to derive a variational error analysis result for the presymplectic variational integrator that is used to discretize the adjoint DAE system. Finally, they discuss the application of adjoint systems in the context of optimal control problems, where they prove a similar result.