📋 Changes in this PR: Equalities, Inequalities and Minimizers, Complex Numbers, Polynomials, Exponentials and Logarithms, Convexity, Infimum and Sums, Trigonometry, Algebra, Derivatives and Taylor Series, Double Integrals and Fubini’s Theorem, Integrals and the Fundamental Theorem of Calculus, Calculus, Dependency Tree, Expressions and Algorithms, Functions, Growth Rates, Math for Data Science, Indicator Functions, Induction and Recursion, Complex Vectors and Matrices, Diagonalization and Cholesky, Direct Sums and Orthogonal Complements, Inner Products and Orthogonality, Matrices, Projections and Linear Systems, Column Space, Null Space and Rank-Nullity, Special Matrices and Decompositions, Subspaces and Rank, Vectors, Linear Algebra, Additivity, Measurable Functions and Integrals, \(L^p\) Spaces, Measures and Probability Measures, Product Measures, Sigma-Algebras, Measures, Notation, Number Systems, Quadratic Minimization under Linear Constraints, Convexity in Several Variables, Critical Points and Optimality Conditions, Gradient Descent, Newton’s Method, Quasi-Newton Methods, Optimization, Proof Writing, Relations, Sets and Functions, Sets, Vector Calculus
💡 Tip: If change highlighting is glitchy, add the no-preview-highlights label to this PR to disable it.

📋 Other Formats: 🎞️ Slides

Optimization

The optimization notes are split across these pages:

• Critical Points and Optimality Conditions: flat points and critical points, Fermat’s theorem, Taylor approximations, and the first- and second-order conditions in several variables.
• Convexity in Several Variables: convex sets and functions, epigraphs, tangent planes, and the first-order and Hessian characterizations of convexity.
• Quadratic Minimization under Linear Constraints: minimization problems, optimization algorithms, the Lagrangian, and the sensitivity of the minimum to the constraints.
• Newton’s Method: iterative methods, the quadratic model, and quadratic convergence.
• Gradient Descent: descent directions, line search, step lengths, and the failure modes of gradient descent on the Rosenbrock function, double wells, and plateaus.
• Quasi-Newton Methods: the secant condition, the BFGS update, and L-BFGS.

A link to a section, theorem or definition on this page from before the split goes on to the page that now holds it.

1 Further reading

• Boyd and Vandenberghe [1] is the standard reference on convex optimization, covering convex sets, convex functions, convex optimization problems, duality, unconstrained minimization, and interior-point methods.
• Nocedal and Wright [2] is a comprehensive reference on continuous numerical optimization, covering line search methods, trust-region methods, conjugate gradient methods, quasi-Newton methods, and nonlinear equations.
• M/CS 335, Linear Optimization at Western Washington University covers optimizing linear functions under linear constraints, including linear programming and duality.
• M/CS 375, Numerical Computation at Western Washington University covers computer arithmetic, solving nonlinear equations, optimization in a single variable, matrix factorization, and iterative methods for matrices.
• Hua Zhou’s course page for “UCLA Biostat 257 - Statistical Computing” (2025 Spring) has lectures on optimization, from root finding to constrained problems. Its source is on GitHub under the MIT License.
• Hua Zhou’s lecture notes for “UCLA Biostat 216 - Mathematical Methods for Biostatistics” (2024 Fall)
• Neural Networks is a YouTube playlist by Grant Sanderson (3Blue1Brown); its chapters on gradient descent and backpropagation calculus visually illustrate how gradients of multivariate cost functions are computed and used for optimization.

References

[1]

S. Boyd and L. Vandenberghe, Convex optimization. Cambridge University Press, 2004. doi: 10.1017/CBO9780511804441.

[2]

J. Nocedal and S. J. Wright, Numerical optimization, 2nd ed. in Springer series in operations research and financial engineering. Springer, 2006. doi: 10.1007/978-0-387-40065-5.