Mathematical Prerequisites
Linear algebra, calculus, and matrix decompositions
Consolidated from Kameron Decker Harris’ 2022 weekly notes and Logan Sizemore’s Spring 2025 math review lecture.
This appendix fixes the notation the rest of the notes use and directs you to the worked calculations, interactive widgets, and exercises in the lab’s shared prerequisite site: Math for Data Science. The probability foundations are in Probability, and the statistical prerequisites are in Statistics.
Everything here is standard. Goodfellow et al. [1], chapter 2, covers linear algebra, and chapter 4 covers derivatives, gradients, and gradient-based optimization. Bishop [2] covers matrix identities in Appendix C, and Boyd and Vandenberghe [3] covers mathematical background in Appendix A.
Almost every model in these notes is built out of a handful of operations on vectors and matrices, together with the derivatives of functions that take those objects as input. Math for Data Science, the lab’s shared mathematics prerequisites, defines each of them with worked examples. This appendix links to those definitions rather than repeating them, and for the main ones says where these notes use them.
None of it is specific to machine learning. For a fuller treatment of the linear algebra, see chapter 2 of Goodfellow, Bengio and Courville [1].
1 Notation
Every model in this course takes numbers in and gives numbers out, and the numbers arrive as vectors (lists of numbers). One email is a vector of word counts; one patient is a vector of measurements. This chapter fixes the notation for those vectors, for the functions that map a vector to a number or to another vector, and for the derivative that tells us which direction to adjust a model.
Three conventions hold throughout the notes:
- A single real number (a scalar) is written in plain italic: \(x\), \(w\), \(b\).
- A column vector of \(\ninputvars\) real numbers, an \(\ninputvars \times 1\) array, is written with
\vx(\(\vx \in \reals^{\ninputvars \times 1}\)) or\vec{x}. Vectors are columns unless something says otherwise. - A matrix — a rectangle of real numbers with \(m\) rows and \(n\) columns — is a capital letter: \(W \in \reals^{m \times n}\), or \(\mX\) for data matrices.
Scalars, vectors and matrices in these notes have real entries, with one exception: a covariate vector \(\vpredvar\) can mix numbers and categories, because a raw covariate may be categorical, such as a car’s country of origin. A feature function encodes it as numbers, which gives its feature vector, a column vector in \(\reals^{\ninputvars \times 1}\).
Other sources write \(\reals^{\ninputvars}\) for \(\reals^{\ninputvars \times 1}\), treat a vector as a row, or allow complex entries or categorical entries in a vector.
\(x_i\) is the \(i\)th entry of \(\vx\), and \(W_{ij}\) is the entry of \(W\) in row \(i\) and column \(j\). Row index first, always.
These notes use the shared mathematics macros from Morrison-Lab/macros, so their notation matches the notation chapters of Math for Data Science, Probability for Data Science, and Regression Models for Epidemiology. We write column vectors with \vx (\(\vx\)), which gives the same typography across all of the lab’s notes repositories.
For logical connectives, quantifiers, set notation, and proofs, see the Math for Data Science Notation page and Probability for Data Science Notation page.
1.1 Where different notation conventions come from
Machine learning, statistics, computer science, and econometrics developed their notations independently before converging on modern data science. Students reading across these literatures encounter different symbols for the same objects. Knowing where these conventions originate explains the choices:
Weights \(\vw, b\) vs. coefficients \(\vbeta, \beta_0\): In artificial intelligence and neural networks [1], [2], a linear function is written \(f(\vx) = \dprod{\vw}{\vx} + b\) (or \(W\vx + \vec{b}\)), where \(\vw\) contains synaptic weights and \(b\) is the bias shifting the activation threshold. In classical statistics and econometrics [4], [5], regression models trace back to Gauss and Legendre, using Greek letters (\(\vbeta\)) for unknown population parameters, and either Roman letters (\(\vec{b}\)) or a hat on the parameter (\(\hvb\)) for sample estimates. This chapter uses \(\vw\) and \(b\) after Goodfellow et al. [1], while Linear Regression and subsequent chapters use the traditional regression notation \(\vcoef\) and \(\beta_0\) (matching Regression Models for Epidemiology and Hastie et al. [4]). They represent the exact same mathematical linear model.
Dimension (\(D\) vs. \(p\)) and sample size (\(N\) vs. \(n\)): Computer science and machine learning texts typically denote the feature dimension by \(D\) (for dimension) or \(d\), with \(N\) data points [1], [2]. Statistical literature traditionally denotes the sample size by \(n\) and the number of predictors by \(p\) (for parameters or predictors) [4], [6].
Targets vs. responses: Neural networks and pattern recognition texts often denote the supervised label by a target \(t\) or \(y\) [2], [7], whereas statistics denotes the outcome as the response or dependent variable \(y\).
Rosetta stones across disciplines: Ripley [7, Sec. 1.1], Table 1.1 provides an early systematic glossary mapping parallel terminology between statistics and neural networks (inputs vs. predictors, outputs vs. responses, weights vs. coefficients, training vs. fitting). In economics, Varian [5] discusses the complementary perspectives and terminology of machine learning and econometrics (such as features vs. regressors and explanatory variables). Breiman [8] and Efron and Hastie [6] provide deeper historical perspective on how algorithmic modeling and data modeling traditions diverged and recombined.
Notation across sources sets our notation beside Brian Hutchinson’s and Kameron Decker Harris’ lecture notes and five widely used texts, one row per concept.
1.2 Vectors and inner products
Math for Data Science covers:
Column vectors and the transpose. Both Math for Data Science and these notes write a vector with a tilde under the letter, using the shared macro \(\vx\) (and transpose \(\vx^\top\)).
The outer product, and the rank of an outer product.
The inner product of two column vectors of the same length multiplies them entry by entry and adds up the results, so it takes two vectors in and returns a single number. Math for Data Science defines it, with a worked example, as the dot product. These notes write it as
\[\dprod{\va}{\vec{b}} = \sumto{i}{\ninputvars} a_i b_i \tag{1}\]
It is also called the dot product or the scalar product [9]. The three names are interchangeable. Which one you meet depends on whether the author came to the subject through linear algebra, through vector calculus, or through machine learning. These notes say inner product.
Almost every model in this course computes a score that way: a vector of weights, one per feature, and a vector of feature values, joined by their inner product.
A vector’s length, or \(\lpnorm{2}\) norm, is the square root of its inner product (Equation 1) with itself. Math for Data Science defines it as the Euclidean norm. These notes write it as
\[ \begin{aligned} \norm{ \va}_2 &= \sqrt{\dprod{\va}{\va}} \\ &= \sqrt{\sumto{i}{\ninputvars} a_i^2} \end{aligned} \tag{2}\]
It is also called a vector’s magnitude. The subscript 2 distinguishes it from the other \(\lpnorm{p}\) norms, and an unsubscripted \(\norm{ \va}\) almost always means this one.
In two dimensions that is the Pythagorean theorem, and in \(\ninputvars\) dimensions it is the same formula with more terms. Norms return when we penalize large weights in Linear Regression.
Math for Data Science provides a worked exercise:
- Sanderson’s video on vectors [10, Ch. 1], and on the dot product as a projection, from 0:51 [10, Ch. 9].
- Hutchinson’s Linear Function Basics (24 min) covers inner products, norms and hyperplanes [11]. The video site requires a login, which enrolled students find in the syllabus.
GeeksforGeeks explains:
1.3 Matrices and the matrix product
Math for Data Science covers:
- Rank.
- The inverse and invertible matrices.
- How matrix arithmetic compares with arithmetic on numbers: products do not commute, a nonzero matrix may have no inverse, and a factor can be cancelled only if it is invertible.
- Diagonal, symmetric and orthogonal matrices, and the fact that an orthogonal matrix preserves length.
- The trace and its cyclic property.
- The matrix inner product, computed entry by entry, and the Frobenius norm. Unsupervised Learning measures the error of a truncated decomposition with the Frobenius norm.
A matrix stores a whole dataset or a whole layer of a model.
Math for Data Science defines the matrix product \(AB\). The product exists only when the inner dimensions agree: \(A\) must be \(m \times n\) and \(B\) must be \(n \times p\), and the result is \(m \times p\). Entry \((i, j)\) of the product is the inner product (Equation 1) of row \(i\) of \(A\) with column \(j\) of \(B\) [14].
Definition 1 (The shape rule) Write the two shapes side by side, and read off
\[(m \times n)(n \times p) \;\longrightarrow\; m \times p \tag{3}\]
The inner pair must match, and the outer pair is the answer’s shape.
The shape rule checks any expression in these notes quickly. If the shapes do not match, the expression is wrong, and we find that out without computing a single number.
Math for Data Science provides a worked exercise:
- Sanderson draws the matrix product as a composition of two linear maps [10, Ch. 4].
- Starmer explains the matrix algebra of neural networks [15].
GeeksforGeeks explains matrix multiplication [16].
2 Linear functions
Every linear model in these notes is built out of linear and affine maps. Math for Data Science covers their definitions, geometry, and exercises:
- Linear functions, which send \(\vzero\) to \(\vzero\).
- Affine functions of the form \(f(\vx) = W\vx + \vec{b}\), and single-output affine functions \(f(\vx) = \dprod{\vw}{\vx} + b\). These notes follow standard machine learning usage and call both linear; Linear Regression keeps the same convention and says so at the same point.
- Lines, planes, and hyperplanes, including an interactive plane tilting widget. Linear Regression defines the general hyperplane.
Sanderson draws linear transformations as matrices [10, Ch. 3].
GeeksforGeeks explains linear transformations [17].
Worked exercises in Math for Data Science:
3 Derivatives and gradients
Fitting a model means adjusting parameters until some measure of its error is small, guided by the gradient. The Gradient Descent chapter uses it. Math for Data Science covers calculus foundations and gradient checks:
- The one-variable derivative and rules for computing derivatives.
- Interactive widget: Linear Approximation Explorer.
- The vector derivative and gradient \(\grad_{\vw} f(\vw)\).
- Checking a gradient by its shape: the shape rule applied to calculus.
- Interactive widget: Gradient Explorer.
- The matrix derivative of a scalar-valued function of a matrix.
Worked exercises in Math for Data Science:
- Finding flat points and critical points
- Computing partial derivatives and gradient vectors by hand
- Finding the gradient of a linear function
- Khan Academy defines the gradient and draws it as a vector field [18].
- Hutchinson’s Gradients Refresher (17 min) covers gradients and partial derivatives [11].
- Starmer explains the chain rule [19].
GeeksforGeeks explains:
4 Decompositions
Math for Data Science covers:
- Eigenvalues and eigenvectors, the spectral theorem and the eigendecomposition of a symmetric matrix. Linear Regression uses the eigendecomposition of \(X^\top X\) to build a usable solution when \(X^\top X\) is not invertible.
- The existence of the singular value decomposition (SVD), the SVD itself, and how an SVD of \(X\) gives an eigendecomposition of \(X^\top X\). Unsupervised Learning truncates the SVD to get principal component analysis.
Math for Data Science writes the SVD as \(\mathbf{U}\mathbf{D}\mathbf{V}^\top\). These notes write \(U S V^\top\) instead, because \(\ninputvars\) is the number of features throughout these notes. Both Math for Data Science and these notes write an eigendecomposition as \(Q \Eigvals Q^\top\) (or \(\mathbf{Q}\mathbf{\Eigvals}\mathbf{Q}^\top\)), where \(Q\) is the orthogonal matrix of eigenvectors [22].
- Sanderson’s video on eigenvectors and eigenvalues [10, Ch. 14], and Brunton’s overview of the SVD and mathematical overview of the SVD [23].
- Avati’s linear-algebra review covers eigenvalues, from 11:15, and quadratic forms, from 23:10 [24], Lecture 2.
GeeksforGeeks explains: