Embedded geometry: second order
Nicolas Boumal · Cambridge University Press eBooks · 2023
To design more sophisticated optimization algorithms, we need more refined geometric tools. In particular, to define the Hessian of a cost function, we need a means to differentiate the gradient vector field. This chapter highlights why this requires care, then proceeds to define connections: the proper concept from differential geometry for this task. The proposed definition is stated somewhat differently from the usual: An optional section details why they are equivalent. Riemannian manifolds have a privileged connection called the Riemannian connection, which is used to define Riemannian Hessians. The same concept is used to differentiate vector fields along curves. Applied to the velocity vector field of a curve, this yields the notion of intrinsic acceleration; geodesics are the curves with zero intrinsic acceleration. The tools built in this chapter naturally lead to second-order Taylor expansions of cost functions along curves. These then motivate the definition of second-order retractions. Two optional closing sections further consider the important special case of Hessians on Riemannian submanifolds, and an intuitive way to build second-order retractions by projection.