Optimization

Last modified: 2026-11-07 12:26:52 (PST)

The optimization notes are split across these pages:

Critical Points and Optimality Conditions: flat points and critical points, Fermat’s theorem, Taylor approximations, and the first- and second-order conditions in several variables.
Convexity in Several Variables: convex sets and functions, epigraphs, tangent planes, and the first-order and Hessian characterizations of convexity.
Quadratic Minimization under Linear Constraints: minimization problems, optimization algorithms, the Lagrangian, and the sensitivity of the minimum to the constraints.
Newton’s Method: iterative methods, the quadratic model, and quadratic convergence.
Gradient Descent: descent directions, line search, step lengths, and the failure modes of gradient descent on the Rosenbrock function, double wells, and plateaus.
Quasi-Newton Methods: the secant condition, the BFGS update, and L-BFGS.

A link to a section, theorem or definition on this page from before the split goes on to the page that now holds it.

1 Further reading

Boyd and Vandenberghe [1] is the standard reference on convex optimization, covering convex sets, convex functions, convex optimization problems, duality, unconstrained minimization, and interior-point methods.
Nocedal and Wright [2] is a comprehensive reference on continuous numerical optimization, covering line search methods, trust-region methods, conjugate gradient methods, quasi-Newton methods, and nonlinear equations.
M/CS 335, Linear Optimization at Western Washington University covers optimizing linear functions under linear constraints, including linear programming and duality.
M/CS 375, Numerical Computation at Western Washington University covers computer arithmetic, solving nonlinear equations, optimization in a single variable, matrix factorization, and iterative methods for matrices.
Hua Zhou’s course page for “UCLA Biostat 257 - Statistical Computing” (2025 Spring) has lectures on optimization, from root finding to constrained problems. Its source is on GitHub under the MIT License.
Hua Zhou’s lecture notes for “UCLA Biostat 216 - Mathematical Methods for Biostatistics” (2024 Fall)
Neural Networks is a YouTube playlist by Grant Sanderson (3Blue1Brown); its chapters on gradient descent and backpropagation calculus visually illustrate how gradients of multivariate cost functions are computed and used for optimization.

References

[1]

S. Boyd and L. Vandenberghe, Convex optimization. Cambridge University Press, 2004. doi: 10.1017/CBO9780511804441.

[2]

J. Nocedal and S. J. Wright, Numerical optimization, 2nd ed. in Springer series in operations research and financial engineering. Springer, 2006. doi: 10.1007/978-0-387-40065-5.