Last modified: 2026-10-27 20:53:54 (PDT)
Definition 1 (Matrix) A matrix [1] of dimensions \(m \times n\) is a rectangular array of \(m \cdot n\) numbers, arranged in \(m\) rows and \(n\) columns:
\[ \mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix} \]
Remark 1 (Matrix entries, and vectors as matrices). The entry in row \(i\) and column \(j\) of \(\mathbf{A}\) is written \(a_{ij}\) or \((\mathbf{A})_{ij}\). For example, for the \(2 \times 3\) matrix
\[ \mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}, \]
\(a_{12} = 2\) and \(a_{23} = 6\).
A column vector of length \(p\) is a \(p \times 1\) matrix, and a row vector of length \(p\) is a \(1 \times p\) matrix. For example, the column vector \((7, 8)\) is a \(2 \times 1\) matrix, and the row vector \([7,\; 8]\) is a \(1 \times 2\) matrix.
Definition 2 (Tensor) A tensor [2] of order \(k \ge 1\) is an array of numbers with \(k\) indices, \[ \mathcal{A} = \mathopen{}\left(a_{i_1 i_2 \cdots i_k}\right)\mathclose{}, \quad i_1 = 1, \ldots, n_1, \ \ldots, \ i_k = 1, \ldots, n_k, \] and its dimensions are \(n_1 \times n_2 \times \cdots \times n_k\).
Example 1 (Vectors, matrices and images as tensors) A vector of length \(p\) is a tensor of order \(1\), and an \(m \times n\) matrix is a tensor of order \(2\). A \(28 \times 28\) grayscale image is a matrix: the entry \(a_{ij}\) is the brightness of the pixel in row \(i\) and column \(j\). A \(28 \times 28\) color image stores three numbers per pixel (red, green, blue), so it is an order-3 tensor with dimensions \(28 \times 28 \times 3\), and \(a_{ijc}\) is the value of color channel \(c\) at the pixel in row \(i\) and column \(j\). It has \(28 \cdot 28 \cdot 3 = 2352\) entries.
Definition 3 (Matrix transpose) The transpose [3] of an \(m \times n\) matrix \(\mathbf{A}\) is the \(n \times m\) matrix \({\mathbf{A}}^{\top}\) obtained by swapping the rows and columns of \(\mathbf{A}\):
\[({\mathbf{A}}^{\top})_{ij} = a_{ji}\]
Example 2 (Transposing a \(2 \times 3\) matrix) For \(\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}\),
\[ {\mathbf{A}}^{\top} = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix}, \]
a \(3 \times 2\) matrix. For instance, entry \((1, 2)\) of the transpose is \(a_{21} = 4\).
Definition 4 (Zero matrix) The \(m \times n\) zero matrix [4] \(\mathbf{0}_{m \times n}\) (or \(\mathbf{0}\) when dimensions are clear from context) has all entries equal to zero:
\[ \mathbf{0}_{m \times n} = \begin{bmatrix} 0 & 0 & \cdots & 0 \\ 0 & 0 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 0 \end{bmatrix} \]
Example 3 (Zero matrices of two shapes) \[ \mathbf{0}_{2 \times 3} = \begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix}, \qquad \mathbf{0}_{3 \times 2} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \\ 0 & 0 \end{bmatrix}. \]
Both have all entries \(0\), but one is \(2 \times 3\) and the other \(3 \times 2\), so they are different matrices.
Definition 5 (Matrix addition) Two matrices \(\mathbf{A}\) and \(\mathbf{B}\) of the same dimensions \(m \times n\) can be added element-wise; their matrix sum [5] is:
\[(\mathbf{A} + \mathbf{B})_{ij} = a_{ij} + b_{ij}\]
Example 4 (Adding matrices, and a sum that is not defined) \[ \begin{aligned} \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} &= \begin{bmatrix} 1 + 0 & 2 + 1 \\ 3 + 1 & 4 + 0 \end{bmatrix} && \text{(add entry by entry)} \\ &= \begin{bmatrix} 1 & 3 \\ 4 & 4 \end{bmatrix}. && \text{(add)} \end{aligned} \]
A \(2 \times 2\) matrix and a \(2 \times 3\) matrix cannot be added: they do not have the same dimensions, so entry \((1, 3)\) of the sum would have no first term.
Theorem 1 (Matrix addition is commutative) For \(m \times n\) matrices \(\mathbf{A}\) and \(\mathbf{B}\), matrix addition is commutative:
\[\mathbf{A} + \mathbf{B} = \mathbf{B} + \mathbf{A}\]
Theorem 2 (Matrix addition is associative) For \(m \times n\) matrices \(\mathbf{A}\), \(\mathbf{B}\), and \(\mathbf{C}\), matrix addition is associative:
\[(\mathbf{A} + \mathbf{B}) + \mathbf{C} = \mathbf{A} + (\mathbf{B} + \mathbf{C})\]
Theorem 3 (Zero matrix is the additive identity) For an \(m \times n\) matrix \(\mathbf{A}\):
\[\mathbf{A} + \mathbf{0}_{m \times n} = \mathbf{A}\]
Theorem 4 (Additive inverse) For any \(m \times n\) matrix \(\mathbf{A}\), the \(m \times n\) matrix \(-\mathbf{A}\) (defined by \((-\mathbf{A})_{ij} = -a_{ij}\)) satisfies:
\[\mathbf{A} + (-\mathbf{A}) = \mathbf{0}_{m \times n}\]
Definition 6 (Scalar multiplication) The scalar multiple [6] of a matrix \(\mathbf{A}\) by a scalar \(c\) is:
\[(c\mathbf{A})_{ij} = c \cdot a_{ij}\]
Example 5 (A scalar multiple) \[ \begin{aligned} 3 \begin{bmatrix} 1 & -2 \\ 0 & 4 \end{bmatrix} &= \begin{bmatrix} 3 \cdot 1 & 3 \cdot(-2) \\ 3 \cdot 0 & 3 \cdot 4 \end{bmatrix} \\ &= \begin{bmatrix} 3 & -6 \\ 0 & 12 \end{bmatrix}. \end{aligned} \]
Definition 7 (Matrix multiplication) The product [7] of an \(m \times k\) matrix \(\mathbf{A}\) and a \(k \times n\) matrix \(\mathbf{B}\) is the \(m \times n\) matrix \(\mathbf{C} = \mathbf{A}\mathbf{B}\) with entries:
\[c_{ij} = \sum_{s=1}^{k} a_{is}\, b_{sj}\]
Remark 2 (Matrix multiplication is not commutative). When \(\mathbf{A}\mathbf{B}\) and \(\mathbf{B}\mathbf{A}\) are both defined, they are usually different. For example, with
\[ \mathbf{A} = \begin{bmatrix} 1 & 2 \\ 0 & 1 \end{bmatrix}, \qquad \mathbf{B} = \begin{bmatrix} 1 & 0 \\ 1 & 1 \end{bmatrix}, \]
\[ \begin{aligned} \mathbf{A}\mathbf{B} &= \begin{bmatrix} 3 & 2 \\ 1 & 1 \end{bmatrix} \\ \mathbf{B}\mathbf{A} &= \begin{bmatrix} 1 & 2 \\ 1 & 3 \end{bmatrix}, \end{aligned} \]
so \(\mathbf{A}\mathbf{B} \neq \mathbf{B}\mathbf{A}\).
Example 6 (Dot product as matrix multiplication) The dot product of two column vectors \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}\) can be written as a matrix product of the row vector \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\) with the column vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}\):
\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}} &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}} \\ &= [x_1,\; x_2,\; \ldots,\; x_p] \begin{bmatrix} \beta_{1} \\ \beta_{2} \\ \vdots \\ \beta_{p} \end{bmatrix} \\ &= x_1\beta_{1} + x_2\beta_{2} + \cdots + x_p \beta_{p} \end{aligned} \]
Theorem 5 (Matrix multiplication is associative) For an \(m \times k\) matrix \(\mathbf{A}\), a \(k \times l\) matrix \(\mathbf{B}\), and an \(l \times n\) matrix \(\mathbf{C}\):
\[ \underbrace{(\mathbf{A}\mathbf{B})\mathbf{C}}_{m \times n} = \underbrace{\mathbf{A}(\mathbf{B}\mathbf{C})}_{m \times n} \]
Proof. Entry \((i, j)\) of each side, from Definition 7:
\[ \begin{aligned} \mathopen{}\left[(\mathbf{A}\mathbf{B})\mathbf{C}\right]\mathclose{}_{ij} &= \sum_{t=1}^{l} (\mathbf{A}\mathbf{B})_{it}\, c_{tj} && \text{(definition of } (\mathbf{A}\mathbf{B})\mathbf{C} \text{)} \\ &= \sum_{t=1}^{l} \mathopen{}\left(\sum_{s=1}^{k} a_{is}\, b_{st}\right)\mathclose{} c_{tj} && \text{(definition of } \mathbf{A}\mathbf{B} \text{)} \\ &= \sum_{s=1}^{k} a_{is} \mathopen{}\left(\sum_{t=1}^{l} b_{st}\, c_{tj}\right)\mathclose{} && \text{(distribute and swap the finite sums)} \\ &= \sum_{s=1}^{k} a_{is}\, (\mathbf{B}\mathbf{C})_{sj} && \text{(definition of } \mathbf{B}\mathbf{C} \text{)} \\ &= \mathopen{}\left[\mathbf{A}(\mathbf{B}\mathbf{C})\right]\mathclose{}_{ij} && \text{(definition of } \mathbf{A}(\mathbf{B}\mathbf{C}) \text{)} \end{aligned} \]
Theorem 6 (Matrix multiplication is distributive over addition) Matrix multiplication distributes over addition from the left and from the right. For an \(m \times k\) matrix \(\mathbf{A}\) and \(k \times n\) matrices \(\mathbf{B}\) and \(\mathbf{C}\):
\[\mathbf{A}(\mathbf{B} + \mathbf{C}) = \mathbf{A}\mathbf{B} + \mathbf{A}\mathbf{C}\]
For \(m \times k\) matrices \(\mathbf{A}\) and \(\mathbf{B}\) and a \(k \times n\) matrix \(\mathbf{C}\):
\[(\mathbf{A} + \mathbf{B})\mathbf{C} = \mathbf{A}\mathbf{C} + \mathbf{B}\mathbf{C}\]
Theorem 7 (Product of two sums of matrices) For \(m \times k\) matrices \(\mathbf{A}\) and \(\mathbf{B}\) and \(k \times n\) matrices \(\mathbf{C}\) and \(\mathbf{D}\),
\[ (\mathbf{A} + \mathbf{B})(\mathbf{C} + \mathbf{D}) = \mathbf{A}\mathbf{C} + \mathbf{A}\mathbf{D} + \mathbf{B}\mathbf{C} + \mathbf{B}\mathbf{D}. \]
Each product keeps its factors in the order they appear on the left, because matrix multiplication is not commutative. So \(\mathbf{A}\mathbf{D}\) and \(\mathbf{B}\mathbf{C}\) are not collected into \(2\mathbf{A}\mathbf{D}\).
Proof. \[ \begin{aligned} (\mathbf{A} + \mathbf{B})(\mathbf{C} + \mathbf{D}) &= (\mathbf{A} + \mathbf{B})\mathbf{C} + (\mathbf{A} + \mathbf{B})\mathbf{D} && \text{(}\href{#thm-matmul-distrib}{\text{Theorem~6}}\text{, from the left)} \\ &= \mathbf{A}\mathbf{C} + \mathbf{B}\mathbf{C} + \mathbf{A}\mathbf{D} + \mathbf{B}\mathbf{D} && \text{(}\href{#thm-matmul-distrib}{\text{Theorem~6}}\text{, from the right, twice)} \\ &= \mathbf{A}\mathbf{C} + \mathbf{A}\mathbf{D} + \mathbf{B}\mathbf{C} + \mathbf{B}\mathbf{D} && \text{(matrix addition is commutative)} \end{aligned} \]
Example 7 (Why the middle terms do not combine) Take \(\mathbf{A} = \mathbf{C} = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}\) and \(\mathbf{B} = \mathbf{D} = \begin{bmatrix} 1 & 0 \\ 1 & 1 \end{bmatrix}\). Then \(\mathbf{A} + \mathbf{B} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\), so
\[ \begin{aligned} (\mathbf{A} + \mathbf{B})^2 &= \begin{bmatrix} 5 & 4 \\ 4 & 5 \end{bmatrix}, \\ \mathbf{A}^2 + \mathbf{A}\mathbf{B} + \mathbf{B}\mathbf{A} + \mathbf{B}^2 &= \begin{bmatrix} 1 & 2 \\ 0 & 1 \end{bmatrix} + \begin{bmatrix} 2 & 1 \\ 1 & 1 \end{bmatrix} + \begin{bmatrix} 1 & 1 \\ 1 & 2 \end{bmatrix} + \begin{bmatrix} 1 & 0 \\ 2 & 1 \end{bmatrix} \\ &= \begin{bmatrix} 5 & 4 \\ 4 & 5 \end{bmatrix}. \end{aligned} \]
Here \(\mathbf{A}\mathbf{B} \neq \mathbf{B}\mathbf{A}\), so the middle terms do not combine into \(2\mathbf{A}\mathbf{B}\). The squared norm of a sum of vectors, \(\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rVert\mathclose{}^2\), is a case where they do combine (Theorem 2 in Inner Products and Orthogonality), because \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\). Matrices that commute, such as \(\mathbf{A}\) and \(\mathbf{A}^2\), combine the same way.
Exercise 1 (Which of these products are defined?) Let \(\mathbf{A}\) be \(4 \times 3\), let \(\mathbf{B}\) be \(3 \times 2\), and let \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^3\). For each expression, say whether it is defined, and if so give the shape of the result:
Solution 1. A vector in \(\mathbb{R}^3\) is a \(3 \times 1\) matrix, so the same rule settles every case.
The last two use the same two vectors and differ only in order, and they return objects of different kinds. Reading a transpose as decoration rather than as a shape change is the commonest way to lose track of an expression.
Definition 8 (Square matrix) A matrix is square [8] if it has the same number of rows as columns.
Example 8 (Square and not square) \(\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}\) is square (\(2\) rows, \(2\) columns); \(\begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}\) is not (\(2\) rows, \(3\) columns).
Definition 9 (Main diagonal) The main diagonal [9] of an \(m \times n\) matrix \(\mathbf{A}\) is the list of its entries \(a_{11}, a_{22}, \ldots, a_{kk}\), where \(k\) is the smaller of \(m\) and \(n\).
Definition 10 (Diagonal entry) The entries on the main diagonal (Definition 9) of a matrix \(\mathbf{A}\) are the diagonal entries of \(\mathbf{A}\).
Definition 11 (Off-diagonal entry) In an \(m \times n\) matrix \(\mathbf{A}\) with main diagonal (Definition 9), every entry \(a_{ij}\) with \(i \ne j\) is an off-diagonal entry.
Example 9 (Diagonal entries of a square and a non-square matrix)
Definition 12 (Identity matrix) The \(p \times p\) identity matrix [10] \(\mathbf{I}_p\) (or \(\mathbf{I}\) when the size is clear from context) has ones on the main diagonal and zeros elsewhere:
\[ (\mathbf{I}_p)_{ij} = \begin{cases} 1 & \text{if } i = j \\ 0 & \text{if } i \neq j \end{cases} \qquad \mathbf{I}_p = \begin{bmatrix} 1 & 0 & \cdots & 0 \\ 0 & 1 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 1 \end{bmatrix} \]
Example 10 (The \(3 \times 3\) identity) \[ \mathbf{I}_3 = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}, \]
so, for instance, \((\mathbf{I}_3)_{22} = 1\) and \((\mathbf{I}_3)_{23} = 0\).
Remark 3 (The identity matrix and the Kronecker delta). The entries of the identity matrix are given by the Kronecker delta: \((\mathbf{I}_p)_{ij} = \delta_{ij}\).
Theorem 8 (Identity matrix is a multiplicative identity) For any \(m \times p\) matrix \(\mathbf{A}\):
\[\mathbf{A}\,\mathbf{I}_p = \mathbf{A}\]
\[\mathbf{I}_m\,\mathbf{A} = \mathbf{A}\]
Definition 13 (Matrix-vector multiplication) The matrix-vector product of an \(m \times p\) matrix \(\mathbf{A}\) and a \(p \times 1\) column vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) is the \(m \times 1\) column vector \(\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) with entries:
\[(\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i = \sum_{j=1}^pa_{ij}\, x_j\]
Remark 4 (Each entry is a dot product). Matrix-vector multiplication extends the dot product (Definition 8 in Vectors): entry \(i\) of \(\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) is the dot product of row \(i\) of \(\mathbf{A}\) with \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\). For example,
\[ \begin{aligned} \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} 1 \\ -1 \end{bmatrix} &= \begin{bmatrix} 1 \cdot 1 + 2 \cdot(-1) \\ 3 \cdot 1 + 4 \cdot(-1) \end{bmatrix} \\ &= \begin{bmatrix} -1 \\ -1 \end{bmatrix}. \end{aligned} \]
Definition 14 (Linear system, system of linear equations) A linear system [11] (or system of linear equations) with \(m\) equations in \(n\) unknowns is an equation \(\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) in which the \(m \times n\) matrix \(\mathbf{A}\) and the vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \in \mathbb{R}^m\) are given. A solution of the system is a vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^n\) for which \(\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) holds.
Example 11 (Two equations in two unknowns) By Definition 13, row \(i\) of \(\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) is the equation \(a_{i1} x_1 + \cdots + a_{in} x_n = b_i\). So the two equations \(x_1 + 2 x_2 = 5\) and \(3 x_1 + 4 x_2 = 6\) form the linear system \(\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \begin{bmatrix} 5 \\ 6 \end{bmatrix}\). The vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (-4, 4.5)\) is a solution:
\[ \begin{aligned} \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} -4 \\ 4.5 \end{bmatrix} &= \begin{bmatrix} 1 \cdot(-4) + 2 \cdot 4.5 \\ 3 \cdot(-4) + 4 \cdot 4.5 \end{bmatrix} && \text{(}\href{#def-matvec-mult}{\text{Definition~13}}\text{)} \\ &= \begin{bmatrix} -4 + 9 \\ -12 + 18 \end{bmatrix} && \text{(multiply)} \\ &= \begin{bmatrix} 5 \\ 6 \end{bmatrix} && \text{(add)} \end{aligned} \]
The vector \((1, 2)\) is not a solution: its first equation holds, since \(1 + 2 \cdot 2 = 5\), but its second gives \(3 \cdot 1 + 4 \cdot 2 = 11 \ne 6\).
Definition 15 (Linear map, linear transformation, linear function) A function \(f: \mathbb{R}^p \to \mathbb{R}^m\) is a linear map [12] (also called a linear transformation or linear function) if it preserves vector addition and scalar multiplication: for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^p\) and \(c \in \mathbb{R}\),
Example 12 (A linear map, and two functions that are not)
\(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (2 x_1,\ x_1 + x_2)\) on \(\mathbb{R}^2\) is linear. For \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^2\) and \(c \in \mathbb{R}\):
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) &= (2 (x_1 + y_1),\ (x_1 + y_1) + (x_2 + y_2)) \\ &= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) + f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) \end{aligned} \]
and
\[ \begin{aligned} f(c \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= (2 c x_1,\ c x_1 + c x_2) \\ &= c f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}). \end{aligned} \]
\(g(x) = x + 1\) on \(\mathbb{R}\) is not linear: additivity fails, since \(g(0 + 0) = 1\) but \(g(0) + g(0) = 2\).
\(h(x) = x^2\) on \(\mathbb{R}\) is not linear: \(h(2 \cdot 1) = 4\) but \(2\,h(1) = 2\), so \(h(c x) \ne c\,h(x)\) for \(c = 2\), \(x = 1\).
Definition 16 (Linear operator) A linear operator on \(\mathbb{R}^p\) is a linear map (Definition 15) from \(\mathbb{R}^p\) to itself, that is, a linear map \(f : \mathbb{R}^p \to \mathbb{R}^p\) whose input and output have the same length. Some authors use “operator” more loosely, for any linear map.
Example 13 (A linear operator, and a linear map that is not one) The map \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (2 x_1,\ x_1 + x_2)\) from Example 12 sends \(\mathbb{R}^2\) to \(\mathbb{R}^2\), so it is a linear operator on \(\mathbb{R}^2\). The linear map \(g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = x_1 + x_2\) sends \(\mathbb{R}^2\) to \(\mathbb{R}\), so it is linear but not an operator: its outputs have length \(1\), not \(2\).
Theorem 9 (A linear map sends zero to zero) If \(f: \mathbb{R}^p \to \mathbb{R}^m\) is a linear map (Definition 15), then \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Proof. On the left, \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) is the zero vector of length \(p\); on the right, it is the zero vector of length \(m\).
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) &= f(0 \cdot\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) && \text{(every entry of } 0 \cdot\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\text{ is } 0 \cdot 0 = 0 \text{; }\href{#def-scalar-mult}{\text{Definition~6}}\text{)} \\&= 0 \cdot f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) && \text{(homogeneity, with } c = 0 \text{ and } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\text{)} \\&= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}} && \text{(every entry of } 0 \cdot f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) \text{ is } 0 \text{; }\href{#def-scalar-mult}{\text{Definition~6}}\text{)} \end{aligned} \]
Example 14 (Ruling out linearity at zero) In Example 12, \(g(x) = x + 1\) has \(g(0) = 1 \neq 0\), so by Theorem 9 it is not linear, without checking additivity or homogeneity.
Sending \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) to \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) is necessary but not sufficient: \(h(x) = x^2\) in the same example has \(h(0) = 0\) and is still not linear.
Theorem 10 (Multiplying by a matrix is a linear map) For any \(m \times p\) matrix \(\mathbf{A}\), the function \(f: \mathbb{R}^p \to \mathbb{R}^m\) defined by \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) is a linear map (Definition 15).
Proof. Both conditions are checked one entry at a time. For each \(i \in \{1, \ldots, m\}\), additivity:
\[ \begin{aligned} \mathopen{}\left(\mathbf{A}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})\right)\mathclose{}_i &= \sum_{j=1}^pa_{ij} (\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})_j && \text{(}\href{#def-matvec-mult}{\text{Definition~13}}\text{)} \\&= \sum_{j=1}^pa_{ij} (x_j + y_j) && \text{(}\href{linear-algebra-vectors.qmd#def-vector-addition}{\text{Definition~7 in Vectors}}\text{)} \\&= \sum_{j=1}^p(a_{ij} x_j + a_{ij} y_j) && \text{(multiplication distributes over addition)} \\&= \sum_{j=1}^pa_{ij} x_j + \sum_{j=1}^pa_{ij} y_j && \text{(a sum of sums splits)} \\&= (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i + (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})_i && \text{(}\href{#def-matvec-mult}{\text{Definition~13}}\text{)} \\&= (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})_i && \text{(}\href{linear-algebra-vectors.qmd#def-vector-addition}{\text{Definition~7 in Vectors}}\text{)} \end{aligned} \]
and homogeneity:
\[ \begin{aligned} \mathopen{}\left(\mathbf{A}(c\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})\right)\mathclose{}_i &= \sum_{j=1}^pa_{ij} (c\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_j && \text{(}\href{#def-matvec-mult}{\text{Definition~13}}\text{)} \\&= \sum_{j=1}^pa_{ij}\, c\, x_j && \text{(}\href{#def-scalar-mult}{\text{Definition~6}}\text{)} \\&= \sum_{j=1}^pc\, a_{ij} x_j && \text{(products are symmetric)} \\&= c \sum_{j=1}^pa_{ij} x_j && \text{(a constant factor comes out of a sum)} \\&= c\, (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i && \text{(}\href{#def-matvec-mult}{\text{Definition~13}}\text{)} \\&= (c\, \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i && \text{(}\href{#def-scalar-mult}{\text{Definition~6}}\text{)} \end{aligned} \]
The four algebra rules cited here are on the algebra page: distributivity, a sum of sums, symmetry of products, and a constant factor.
Theorem 11 (A linear map preserves linear combinations) If \(f: \mathbb{R}^p \to \mathbb{R}^m\) is a linear map (Definition 15), then for any vectors \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1, \ldots, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_k \in \mathbb{R}^p\) and any coefficients \(c_1, \ldots, c_k \in \mathbb{R}\):
\[f\mathopen{}\left(\sum_{j=1}^{k} c_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j\right)\mathclose{} = \sum_{j=1}^{k} c_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j)\]
Proof. By induction on \(k\). For \(k = 1\), the claim \(f(c_1 \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1) = c_1 f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1)\) is homogeneity.
Suppose the claim holds for \(k\) vectors. Then for \(k + 1\):
\[ \begin{aligned} f\mathopen{}\left(\sum_{j=1}^{k+1} c_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j\right)\mathclose{} &= f\mathopen{}\left(\sum_{j=1}^{k} c_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j + c_{k+1} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_{k+1}\right)\mathclose{} && \text{(split off the last term)} \\&= f\mathopen{}\left(\sum_{j=1}^{k} c_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j\right)\mathclose{} + f(c_{k+1} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_{k+1}) && \text{(additivity)} \\&= \sum_{j=1}^{k} c_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j) + f(c_{k+1} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_{k+1}) && \text{(induction hypothesis)} \\&= \sum_{j=1}^{k} c_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j) + c_{k+1} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_{k+1}) && \text{(homogeneity)} \\&= \sum_{j=1}^{k+1} c_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j) && \text{(join the last term back on)} \end{aligned} \]
Theorem 12 (Every linear map is multiplication by a matrix) If \(f: \mathbb{R}^p \to \mathbb{R}^m\) is a linear map (Definition 15), then there is exactly one \(m \times p\) matrix \(\mathbf{A}\) such that
\[f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\quad \text{for all } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\]
Column \(j\) of \(\mathbf{A}\) is \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\), the value of \(f\) at the \(j\)th indicator vector (Definition 18 in Vectors); that is, \(a_{ij} = \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i\).
Proof. Existence. Let \(\mathbf{A}\) be the \(m \times p\) matrix with \(a_{ij} = \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i\). For any \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\),
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= f\mathopen{}\left(\sum_{j=1}^px_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\right)\mathclose{} && \text{(}\href{linear-algebra-vectors.qmd#thm-standard-basis-expansion}{\text{Theorem~4 in Vectors}}\text{)} \\&= \sum_{j=1}^px_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j) && \text{(}\href{#thm-linear-map-lincom}{\text{Theorem~11}}\text{)} \end{aligned} \]
and entry \(i\) of that sum is
\[ \begin{aligned} \mathopen{}\left(\sum_{j=1}^px_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i &= \sum_{j=1}^px_j \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i && \text{(}\href{linear-algebra-vectors.qmd#def-linear-combination}{\text{Definition~9 in Vectors}}\text{, entry by entry)} \\&= \sum_{j=1}^px_j a_{ij} && \text{(definition of } \mathbf{A} \text{)} \\&= \sum_{j=1}^pa_{ij} x_j && \text{(products are symmetric)} \\&= (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i && \text{(}\href{#def-matvec-mult}{\text{Definition~13}}\text{)} \end{aligned} \]
Uniqueness. Suppose \(\mathbf{B}\) is any \(m \times p\) matrix with \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\). Taking \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\), for each \(i\):
\[ \begin{aligned} \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i &= (\mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)_i && \text{(assumption on } \mathbf{B} \text{)} \\&= \sum_{k=1}^{p} b_{ik} (\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)_k && \text{(}\href{#def-matvec-mult}{\text{Definition~13}}\text{)} \\&= b_{ij} \cdot 1 + \sum_{k \neq j} b_{ik} \cdot 0 && \text{(}\href{linear-algebra-vectors.qmd#def-indicator-vector}{\text{Definition~18 in Vectors}}\text{, splitting off the } k = j \text{ term)} \\&= b_{ij} + 0 && \text{(times } 1 \text{ changes nothing; times } 0 \text{ gives } 0 \text{)} \\&= b_{ij} && \text{(adding } 0 \text{ changes nothing)} \end{aligned} \]
So
\[ \begin{aligned} b_{ij} &= \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i \\ &= a_{ij} \end{aligned} \]
for every \(i\) and \(j\), and \(\mathbf{B} = \mathbf{A}\).
Definition 17 (Matrix of a linear map) The matrix of a linear map \(f: \mathbb{R}^p \to \mathbb{R}^m\) is the unique \(m \times p\) matrix \(\mathbf{A}\) with \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) (Theorem 12). Its column \(j\) is \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\).
Example 15 (Finding the matrix of a linear map) For \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (2 x_1,\ x_1 + x_2)\) from Example 12, the values of \(f\) at the indicator vectors are
\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_1) = f\mathopen{}\left(\begin{bmatrix}1 \\ 0\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}2 \cdot 1 \\ 1 + 0\end{bmatrix} = \begin{bmatrix}2 \\ 1\end{bmatrix}, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_2) = f\mathopen{}\left(\begin{bmatrix}0 \\ 1\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}2 \cdot 0 \\ 0 + 1\end{bmatrix} = \begin{bmatrix}0 \\ 1\end{bmatrix} \]
and these are the columns of the matrix of \(f\) (Definition 17): \(\mathbf{A} = \begin{bmatrix}2 & 0 \\ 1 & 1\end{bmatrix}\).
Corollary 1 (Linear functions from vectors to numbers) A function \(f: \mathbb{R}^p \to \mathbb{R}\) is linear (Definition 15) if and only if there is a vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \in \mathbb{R}^p\) such that
\[f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} \quad \text{for all } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\]
and then \(w_j = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\) for each \(j\).
Proof. If \(f\) is linear. Theorem 12 with \(m = 1\) gives a \(1 \times p\) matrix \(\mathbf{A}\) with \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and \(a_{1j} = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\). Let \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\) be the vector with \(w_j = a_{1j}\), so that \(\mathbf{A} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top}\). Then
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(}\href{#thm-linear-map-matrix}{\text{Theorem~12}}\text{)} \\&= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && (\mathbf{A} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top}) \end{aligned} \]
and
\[ \begin{aligned} w_j &= a_{1j} \\ &= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j). \end{aligned} \]
If \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\). Here \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top}\) is a \(1 \times p\) matrix, so \(f\) is linear by Theorem 10.
Example 16 (A linear function from \(\mathbb{R}^2\) to \(\mathbb{R}\)) \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 3 x_1 + 5 x_2\) is \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) with \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} = {(3, 5)}^{\top}\), so it is linear by Corollary 1. The entries of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\) are the values at the indicator vectors:
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_1) &= 3 \cdot 1 + 5 \cdot 0 \\ &= 3 \end{aligned} \]
and
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_2) &= 3 \cdot 0 + 5 \cdot 1 \\ &= 5. \end{aligned} \]
\(g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 3 x_1 + 5 x_2 + 1\) is not linear, because \(g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = 1 \neq 0\) (Theorem 9).
Videos
Two chapters of the Essence of linear algebra series by 3Blue1Brown show the results above geometrically:
Definition 18 (Translation) For a fixed vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \in \mathbb{R}^m\), the translation [13] by \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) is the function \(t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}} : \mathbb{R}^m \to \mathbb{R}^m\) defined by
\[t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\]
Example 17 (Translating two points) The translation by \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = {(1, -2)}^{\top}\) sends
\[ t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}\mathopen{}\left(\begin{bmatrix}0 \\ 0\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}0 + 1 \\ 0 + (-2)\end{bmatrix} = \begin{bmatrix}1 \\ -2\end{bmatrix}, \qquad t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}\mathopen{}\left(\begin{bmatrix}3 \\ 4\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}3 + 1 \\ 4 + (-2)\end{bmatrix} = \begin{bmatrix}4 \\ 2\end{bmatrix} \]
Both points move one unit right and two units down.
Remark 5 (A translation moves every point by the same vector). A translation by \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) moves every point by the same vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\).
Definition 19 (Affine map, affine transformation, affine function) A function \(f: \mathbb{R}^p \to \mathbb{R}^m\) is an affine map (also called an affine transformation or affine function) if it is a matrix product followed by a translation (Definition 18) (a composition, with the matrix product as the inner function): there are an \(m \times p\) matrix \(\mathbf{A}\) and a vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \in \mathbb{R}^m\) such that
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}(\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) \\ &= \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \quad \text{for all } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p \end{aligned} \]
Definition 20 (Offset of an affine map, intercept, bias) In Definition 19, the vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) is called the offset of \(f\) (statistics also calls it the intercept, and machine learning the bias).
Remark 6 (Real-valued affine maps). For a real-valued function (Definition 4 in Vectors, \(m = 1\)), \(\mathbf{A}\) is a single row, written \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top}\), and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) is a single number \(b\), so an affine function is a linear function \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) (Corollary 1) plus an offset:
\[f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b, \qquad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \in \mathbb{R}^p, \quad b \in \mathbb{R} \tag{1}\]
Definition 21 (Augmented vector) The augmented vector of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\) is the vector \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} \in \mathbb{R}^{p+1}\) formed by placing a \(1\) above the entries of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\):
\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} &= \begin{bmatrix}1 \\ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\end{bmatrix} \\ &= {(1, x_1, \ldots, x_p)}^{\top} \end{aligned} \]
Example 18 (An affine function as a dot product with an augmented vector) For \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= {(2, 5)}^{\top}\), the augmented vector is \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} = {(1, 2, 5)}^{\top}\). With \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} = {(3, -1)}^{\top}\) and \(b = 4\), the affine function \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b\) (Equation 1) equals the dot product of \({(b, w_1, w_2)}^{\top} = {(4, 3, -1)}^{\top}\) with \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}}\):
\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b &= 3 \cdot 2 + (-1) \cdot 5 + 4 \\ &= 5, \\ {(4, 3, -1)}^{\top} \cdot {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} &= 4 \cdot 1 + 3 \cdot 2 + (-1) \cdot 5 \\ &= 5 \end{aligned} \]
Theorem 13 (An affine map sends zero to its offset) An affine map \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) (Definition 19) satisfies \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\).
Proof. First, \(\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\): for each \(i\),
\[ \begin{aligned} (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}})_i &= \sum_{j=1}^pa_{ij} \cdot 0 && \text{(}\href{#def-matvec-mult}{\text{Definition~13}}\text{)} \\&= \sum_{j=1}^p0 && \text{(any number times } 0 \text{ is } 0 \text{)} \\&= 0 && \text{(a sum of zeros is } 0 \text{)} \end{aligned} \]
Then
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} && \text{(}\href{#def-affine-map}{\text{Definition~19}}\text{)} \\&= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} && (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) \\&= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} && \text{(adding } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\text{ changes no entry; }\href{linear-algebra-vectors.qmd#def-vector-addition}{\text{Definition~7 in Vectors}}\text{)} \end{aligned} \]
Theorem 14 (An affine map is linear exactly when its offset is zero) An affine map \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) (Definition 19) is linear (Definition 15) if and only if \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Proof. If \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), then
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\\ &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \end{aligned} \]
which is linear by Theorem 10. Conversely, if \(f\) is linear, then \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) by Theorem 9, and \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) by Theorem 13, so \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Example 19 (Two affine maps with the same matrix) Let \(\mathbf{A} = \begin{bmatrix}2 & 0 \\ 1 & 1\end{bmatrix}\), the matrix of the linear map in Example 12. \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) has offset \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), so it is linear. \(g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \begin{bmatrix}1 \\ 0\end{bmatrix}\) has offset \(\begin{bmatrix}1 \\ 0\end{bmatrix} \neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), so by Theorem 14 it is not linear: indeed \(g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \begin{bmatrix}1 \\ 0\end{bmatrix}\), not \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Corollary 2 (Every linear map is affine) Every linear map \(f : \mathbb{R}^p \to \mathbb{R}^m\) (Definition 15) is an affine map (Definition 19) with offset \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Proof. If \(f\) is linear, Theorem 12 gives a matrix \(\mathbf{A}\) with
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\\ &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}, \end{aligned} \]
which is Definition 19 with \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Theorem 15 (An affine map determines its matrix and offset) If \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) and \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}'\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}'\) for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\) (Definition 19), then \(\mathbf{A}' = \mathbf{A}\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}' = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\).
Proof. By Theorem 13, applied to each formula,
\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} &= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) \\ &= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}'. \end{aligned} \]
Then
\[ \begin{aligned} \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}&= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) - \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \\ &= \mathbf{A}'\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} \end{aligned} \]
for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\), so \(\mathbf{A}\) and \(\mathbf{A}'\) are both matrices (Definition 17) of the linear map \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\mapsto f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) - \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) (linear by Theorem 10), and the uniqueness in Theorem 12 gives \(\mathbf{A}' = \mathbf{A}\).
Remark 7 (“Linear” in this site, and in machine learning and statistics). This site uses “linear” only in the sense of Definition 15, so by Theorem 14 a function with a nonzero offset is affine but not linear. For functions of one variable, this is the same split as affine \(a x + b\) versus linear \(a x\) on the algebra page. (The algebra page writes these as \(m x + b\) and \(m x\); here \(m\) already names the number of outputs.)
Machine learning and statistics often call any function of the form Equation 1 “linear”, as in a linear model in statistics or a linear layer in a neural network, and call the offset \(b\) the bias (machine learning) or the intercept (statistics), as Definition 19 notes. A model with an intercept is affine in \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) but linear in its coefficients \((b, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}})\), and that is the sense of “linear” in “linear model”: it stays linear in the coefficients even when the inputs are transformed: \(w_1 x + w_2 x^2 + b\) is not affine in \(x\), but it is linear in \((b, w_1, w_2)\). Separately, the affine function \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b\) is a linear function of the augmented vector \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}}\) (Definition 21), with \(b\) as the coefficient on its first entry.
Further reading
Definition 22 (Hyperplane) A hyperplane [16] in \(\mathbb{R}^p\) is the set of points \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\) satisfying
\[\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b = 0\]
for some vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \in \mathbb{R}^p \setminus \{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\}\) and number \(b \in \mathbb{R}\). The number \(b\) is the offset of the hyperplane.
Definition 23 (Normal vector of a hyperplane) In Definition 22, the vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\) is a normal vector [17] of the hyperplane.
Example 20 (A line in \(\mathbb{R}^2\), and why the normal vector must be nonzero)
viewof planeW1 = Inputs.range([-1.5, 1.5], {value: 1, step: 0.1, label: "w1"})
viewof planeW2 = Inputs.range([-1.5, 1.5], {value: 0.5, step: 0.1, label: "w2"})
viewof planeB = Inputs.range([-2, 2], {value: 0, step: 0.1, label: "b"})
viewof planeTurn = Inputs.range([0, 360], {value: 30, step: 5, label: "turn the view"}){
const sign = (v) => (v < 0 ? "\u2212" : "+");
const fmt = (v) => Math.abs(v).toFixed(1);
return md`f(x1, x2) = ${planeW1.toFixed(1)} x1 ${sign(planeW2)} ${fmt(planeW2)} x2 ${sign(planeB)} ${fmt(planeB)}.
The orange slice, along x1 with x2 = 0, rises ${planeW1.toFixed(1)} for each unit of x1;
the green slice, along x2, rises ${planeW2.toFixed(1)} for each unit of x2.
Both slices cross the vertical axis at height b = ${planeB.toFixed(1)}.`;
}// The graph z = f(x1, x2) over the square [-2, 2] x [-2, 2], drawn in 3-D:
// turn the square about the vertical axis, then look down on it from 30 degrees.
planeView = {
const f = (x1, x2) => planeW1 * x1 + planeW2 * x2 + planeB;
const w = 340, h = 300, s = 38, sz = 16, elev = 30 * Math.PI / 180;
const turn = planeTurn * Math.PI / 180;
const at = (x1, x2, z) => {
const u = x1 * Math.cos(turn) - x2 * Math.sin(turn);
const v = x1 * Math.sin(turn) + x2 * Math.cos(turn);
return [w / 2 + s * u, h / 2 - s * Math.sin(elev) * v - sz * Math.cos(elev) * z];
};
const path = (pts) => d3.line()(pts.map(([a, b, z]) => at(a, b, z)));
const svg = d3.create("svg").attr("width", w).attr("height", h)
.attr("role", "img")
.attr("aria-label", "A tilted plane drawn in three dimensions over a square of inputs, " +
"with its two slices through the origin highlighted, " +
"and a vertical marker at the origin showing the height b.");
const ticks = d3.range(-2, 2.01, 0.5);
const corners = [[-2, -2], [2, -2], [2, 2], [-2, 2], [-2, -2]];
// the floor, z = 0, and the three axes
svg.append("path").attr("d", path(corners.map(([a, b]) => [a, b, 0])))
.attr("fill", "#eee").attr("stroke", "#bbb");
const axis = (p, q, name) => {
svg.append("path").attr("d", path([p, q])).attr("stroke", "#888");
const [tx, ty] = at(...q);
svg.append("text").attr("x", tx + 4).attr("y", ty).attr("font-size", 13).text(name);
};
axis([0, 0, 0], [2.6, 0, 0], "x1");
axis([0, 0, 0], [0, 2.6, 0], "x2");
axis([0, 0, -8], [0, 0, 8], "f");
// the plane itself, as a mesh of lines along each input
svg.append("path").attr("d", path(corners.map(([a, b]) => [a, b, f(a, b)])))
.attr("fill", "#1f77b4").attr("fill-opacity", 0.18).attr("stroke", "#1f77b4");
for (const t of ticks) {
svg.append("path").attr("d", path([[t, -2, f(t, -2)], [t, 2, f(t, 2)]]))
.attr("stroke", "#1f77b4").attr("stroke-opacity", 0.35).attr("fill", "none");
svg.append("path").attr("d", path([[-2, t, f(-2, t)], [2, t, f(2, t)]]))
.attr("stroke", "#1f77b4").attr("stroke-opacity", 0.35).attr("fill", "none");
}
// the slice along x1 (x2 = 0) has slope w1; the slice along x2 has slope w2
svg.append("path").attr("d", path([[-2, 0, f(-2, 0)], [2, 0, f(2, 0)]]))
.attr("stroke", "#ff7f0e").attr("stroke-width", 3);
svg.append("path").attr("d", path([[0, -2, f(0, -2)], [0, 2, f(0, 2)]]))
.attr("stroke", "#2ca02c").attr("stroke-width", 3);
const [ox, oy] = at(0, 0, planeB);
svg.append("circle").attr("cx", ox).attr("cy", oy).attr("r", 4).attr("fill", "#222");
svg.append("text").attr("x", ox + 6).attr("y", oy - 6).attr("font-size", 13).text("b");
return svg.node();
}Plot.plot({
ariaLabel: 'Contour map of the same plane seen from directly above, ' +
'with x1 across and x2 up, ' +
'shaded by the value of f, ' +
'with straight level lines one unit of f apart.',
width: 300, height: 300, marginLeft: 40,
x: {domain: [-2, 2], label: "x1"},
y: {domain: [-2, 2], label: "x2"},
color: {type: "diverging", scheme: "RdBu", domain: [-8, 8], legend: true, label: "f(x1, x2)"},
marks: [
Plot.contour({
x1: -2, y1: -2, x2: 2, y2: 2,
fill: (x1, x2) => planeW1 * x1 + planeW2 * x2 + planeB,
thresholds: d3.range(-8, 8.01, 1), stroke: "#333", strokeOpacity: 0.5
}),
Plot.ruleY([0], {stroke: "#ff7f0e", strokeWidth: 3}),
Plot.ruleX([0], {stroke: "#2ca02c", strokeWidth: 3})
]
})Exercise 2 (Write a rule in matrix form) A clinic scores a patient’s risk from three measurements.
| symbol | meaning | units |
|---|---|---|
| \(x_1\) | age | years |
| \(x_2\) | systolic blood pressure | mmHg |
| \(x_3\) | body mass index | kg/m\(^2\) |
The rule in use is
\[\text{score} = 0.4\,x_1 + 0.2\,x_2 + 1.5\,x_3 - 30\]
Write this rule in the form of Equation 1, naming \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\), \(b\) and \(p\) explicitly.
Solution 2. Read the coefficients straight off the rule. There are three inputs, so \(p = 3\):
\[\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} = \begin{bmatrix} 0.4 \\ 0.2 \\ 1.5 \end{bmatrix}, \qquad b = -30\]
and the rule is \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ b\).
Exercise 3 (Score a patient) Give the score from the rule in Exercise 2 for a patient aged \(50\) with blood pressure \(130\) and BMI \(28\).
Solution 3. For the patient,
\[\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \begin{bmatrix} 50 \\ 130 \\ 28 \end{bmatrix}\]
\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= (0.4)(50) + (0.2)(130) + (1.5)(28) \\ &= 20 + 26 + 42 \\ &= 88 \end{aligned} \]
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= 88 - 30 \\ &= 58 \end{aligned} \]
Two things are worth noticing. The coefficients are not comparable as they stand. The weight on BMI, \(1.5\), is \(7.5\) times the weight on blood pressure, \(0.2\), but the contribution of BMI, \(42\), is less than twice the contribution of blood pressure, \(26\), because the BMI value, \(28\), is much smaller than the blood pressure value, \(130\). So the weight on a feature says nothing on its own about how much that feature contributes to the score. And \(b = -30\) changes the score: without it, the score would be \(88\) instead of \(58\).
Exercise 4 (Which of these are affine in \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\)?) Each of the five expressions is a function of the weights \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \in \mathbb{R}^p\), with the data \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and \(y\) held fixed. Say which ones are affine (Definition 19), that is, have the form Equation 1 — an inner product plus an offset — and which of those are also linear (Definition 15).
Solution 4.
Items 1, 2 and 5 are affine and items 3 and 4 are not, which is the split that matters: a model can be affine in its weights while the quantity we minimize is not. Squared error and the norm penalty are both curved in \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\), and that curvature is what makes fitting an optimization problem at all: there is always a downhill direction to follow. Curvature alone does not promise a single answer. The squared error of one observation is completely flat along a whole line of weight vectors, every one of which fits that observation exactly, and pinning down one of them is part of what the norm penalty is added for in week 3.
Theorem 16 (Transpose of a sum) For \(m \times n\) matrices \(\mathbf{A}\) and \(\mathbf{B}\):
\[{(\mathbf{A} + \mathbf{B})}^{\top} = {\mathbf{A}}^{\top} + {\mathbf{B}}^{\top}\]
In particular, for column vectors \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\) of the same length:
\[{(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})}^{\top} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} + {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}}^{\top}\]
Proof. Entry \((i, j)\) of each side:
\[ \begin{aligned} \mathopen{}\left[{(\mathbf{A} + \mathbf{B})}^{\top}\right]\mathclose{}_{ij} &= (\mathbf{A} + \mathbf{B})_{ji} && \text{(definition of the transpose)} \\ &= a_{ji} + b_{ji} && \text{(definition of matrix addition)} \\ &= ({\mathbf{A}}^{\top})_{ij} + ({\mathbf{B}}^{\top})_{ij} && \text{(definition of the transpose)} \\ &= \mathopen{}\left[{\mathbf{A}}^{\top} + {\mathbf{B}}^{\top}\right]\mathclose{}_{ij} && \text{(definition of matrix addition)} \end{aligned} \]
Theorem 17 (Transpose of a product) For an \(m \times k\) matrix \(\mathbf{A}\) and a \(k \times n\) matrix \(\mathbf{B}\):
\[ \underbrace{{(\mathbf{A}\mathbf{B})}^{\top}}_{n \times m} = \underbrace{{\mathbf{B}}^{\top}}_{n \times k}\,\underbrace{{\mathbf{A}}^{\top}}_{k \times m} \]
Proof. Entry \((i, j)\) of each side:
\[ \begin{aligned} \mathopen{}\left[{(\mathbf{A}\mathbf{B})}^{\top}\right]\mathclose{}_{ij} &= (\mathbf{A}\mathbf{B})_{ji} && \text{(definition of the transpose)} \\ &= \sum_{s=1}^{k} a_{js}\, b_{si} && \text{(definition of matrix multiplication)} \\ &= \sum_{s=1}^{k} ({\mathbf{B}}^{\top})_{is}\, ({\mathbf{A}}^{\top})_{sj} && \text{(definition of the transpose; reorder each product)} \\ &= \mathopen{}\left[{\mathbf{B}}^{\top}\,{\mathbf{A}}^{\top}\right]\mathclose{}_{ij} && \text{(definition of matrix multiplication)} \end{aligned} \]
Remark 8 (The order of the factors reverses). Transposing a product reverses the order of the factors. Keeping the original order usually gives an undefined product or a different matrix. For example, with \(\mathbf{A} = [1,\; 2]\) (\(1 \times 2\)) and \(\mathbf{B} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}\) (\(2 \times 1\)),
\[ \begin{aligned} \mathbf{A}\mathbf{B} &= 1 \cdot 3 + 2 \cdot 4 \\ &= 11, \end{aligned} \]
so \({(\mathbf{A}\mathbf{B})}^{\top} = 11\), and
\[ {\mathbf{B}}^{\top}\,{\mathbf{A}}^{\top} = [3,\; 4] \begin{bmatrix} 1 \\ 2 \end{bmatrix} = 11, \qquad {\mathbf{A}}^{\top}\,{\mathbf{B}}^{\top} = \begin{bmatrix} 1 \\ 2 \end{bmatrix} [3,\; 4] = \begin{bmatrix} 3 & 4 \\ 6 & 8 \end{bmatrix}. \]
If instead \(\mathbf{A}\) is \(2 \times 3\) and \(\mathbf{B}\) is \(3 \times 1\), then \({\mathbf{B}}^{\top}\,{\mathbf{A}}^{\top}\) is a \((1 \times 3)(3 \times 2)\) product, but \({\mathbf{A}}^{\top}\,{\mathbf{B}}^{\top}\) is a \((3 \times 2)(1 \times 3)\) product, which is undefined.
Theorem 18 (A \(1 \times 1\) product equals its own transpose) If \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) are vectors of length \(p\) and \(\mathbf{M}\) is a \(p \times p\) matrix, then \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) is a number, and
\[ {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,{\mathbf{M}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}. \]
In particular, \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}\).
Proof. The product \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) is a \((1 \times p)(p \times p)(p \times 1)\) product, so it is a \(1 \times 1\) matrix. The transpose of a \(1 \times 1\) matrix is itself, so
\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} &= {\mathopen{}\left({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\right)\mathclose{}}^{\top} && \text{(a } 1 \times 1 \text{ matrix equals its transpose)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,{\mathbf{M}}^{\top}\,{\mathopen{}\left({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\right)\mathclose{}}^{\top} && \text{(transpose of a product, twice)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,{\mathbf{M}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} && \text{(transposing twice changes nothing)} \end{aligned} \]
The transpose step is Theorem 17. Taking \(\mathbf{M} = \mathbf{I}\) gives the last statement.
Example 21 (Two ways to write the same number) Take \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} = (1, 2)\), \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = (3, -1)\) and \(\mathbf{M} = \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix}\). Then \(\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = (1, -3)\), so
\[ {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = 1 \cdot 1 + 2 \cdot (-3) = -5. \]
Also \({\mathbf{M}}^{\top} = \begin{bmatrix} 1 & 0 \\ 2 & 3 \end{bmatrix}\) and \({\mathbf{M}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} = (1, 8)\), so
\[ {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,{\mathbf{M}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} = 3 \cdot 1 + (-1) \cdot 8 = -5. \]
The two products are equal, as the theorem says. In least squares, this is why the two cross terms of \({(\mathbf{X}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}- \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})}^{\top}(\mathbf{X}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}- \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})\) are equal and can be combined.