Optimization

Last modified: 2026-11-03 15:17:03 (PST)

The optimization notes are split across these pages:

• Critical Points and Optimality Conditions: flat points and critical points, Fermat’s theorem, Taylor approximations, and the first- and second-order conditions in several variables.
• Convexity in Several Variables: convex sets and functions, epigraphs, tangent planes, and the first-order and Hessian characterizations of convexity.
• Quadratic Minimization under Linear Constraints: minimization problems, optimization algorithms, the Lagrangian, and the sensitivity of the minimum to the constraints.
• Newton’s Method: iterative methods, the quadratic model, and quadratic convergence.
• Gradient Descent: descent directions, line search, step lengths, and the failure modes of gradient descent on the Rosenbrock function, double wells, and plateaus.
• Quasi-Newton Methods: the secant condition, the BFGS update, and L-BFGS.

A link to a section, theorem or definition on this page from before the split goes on to the page that now holds it.

1 Further reading

• Boyd and Vandenberghe [1] is the standard reference on convex optimization, covering convex sets, convex functions, convex optimization problems, duality, unconstrained minimization, and interior-point methods.
• Nocedal and Wright [2] is a comprehensive reference on continuous numerical optimization, covering line search methods, trust-region methods, conjugate gradient methods, quasi-Newton methods, and nonlinear equations.
• M/CS 335, Linear Optimization at Western Washington University covers optimizing linear functions under linear constraints, including linear programming and duality.
• M/CS 375, Numerical Computation at Western Washington University covers computer arithmetic, solving nonlinear equations, optimization in a single variable, matrix factorization, and iterative methods for matrices.
• Hua Zhou’s course page for “UCLA Biostat 257 - Statistical Computing” (2025 Spring) has lectures on optimization, from root finding to constrained problems. Its source is on GitHub under the MIT License.
• Hua Zhou’s lecture notes for “UCLA Biostat 216 - Mathematical Methods for Biostatistics” (2024 Fall)
• Neural Networks is a YouTube playlist by Grant Sanderson (3Blue1Brown); its chapters on gradient descent and backpropagation calculus visually illustrate how gradients of multivariate cost functions are computed and used for optimization.

References

[1]

S. Boyd and L. Vandenberghe, Convex optimization. Cambridge University Press, 2004. doi: 10.1017/CBO9780511804441.

[2]

J. Nocedal and S. J. Wright, Numerical optimization, 2nd ed. in Springer series in operations research and financial engineering. Springer, 2006. doi: 10.1007/978-0-387-40065-5.