Matrices

Published

Last modified: 2026-11-13 13:00:00 (PST)

Definition 1 (Matrix) A matrix [1] of dimensions $m \times n$ is a rectangular array of $m \cdot n$ numbers, arranged in $m$ rows and $n$ columns:

\[ \mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix} \]

Remark 1 (Matrix entries, and vectors as matrices)

Remark 1 (Matrix entries, and vectors as matrices). The entry in row $i$ and column $j$ of $\mathbf{A}$ is written $a_{ij}$ or $(\mathbf{A})_{ij}$. For example, for the $2 \times 3$ matrix

\[ \mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}, \]

$a_{12} = 2$ and $a_{23} = 6$.

A column vector of length $p$ is a $p \times 1$ matrix, and a row vector of length $p$ is a $1 \times p$ matrix. For example, the column vector $(7, 8)$ is a $2 \times 1$ matrix, and the row vector $[7,\; 8]$ is a $1 \times 2$ matrix.

Definition 2 (Equal matrices)

Definition 2 (Equal matrices) Two matrices $\mathbf{A}$ and $\mathbf{B}$ are equal, written $\mathbf{A} = \mathbf{B}$, if both are $m \times n$ and $a_{ij} = b_{ij}$ for every $i = 1, \ldots, m$ and $j = 1, \ldots, n$.

Example 1 (Equal and unequal matrices)

Example 1 (Equal and unequal matrices) \[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}, \qquad \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \ne \begin{bmatrix} 1 & 2 \\ 3 & 5 \end{bmatrix}, \qquad \begin{bmatrix} 1 & 2 & 3 & 4 \end{bmatrix} \ne \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}. \]

The second pair differs in one entry. The third pair lists the same four numbers in the same order, but one matrix is $1 \times 4$ and the other $2 \times 2$, so the two are not equal.

Exercise 1 (Solving for an unknown entry)

Exercise 1 (Solving for an unknown entry) Two matrices are equal only when every pair of matching entries agrees, so one matrix equation stands for several equations between numbers. Let

\[ \begin{bmatrix} 2 & x + 1 \\ 0 & 5 \end{bmatrix} = \begin{bmatrix} 2 & 4 \\ 0 & 5 \end{bmatrix}. \]

Find $x$.

Hint

Both matrices are $2 \times 2$, so by Definition 2 each entry on the left is the same number as the entry in the same position on the right.

Solution 1

Solution 1. Both matrices are $2 \times 2$, so they are equal when their entries in each position agree. Three positions already agree, and the entries in position $(1, 2)$ give

\[ \begin{aligned} x + 1 &= 4 && \text{(}\href{#def-matrix-equality}{\text{Definition~2}}\text{)} \\ x &= 3. && \text{(subtract } 1 \text{ from both sides)} \end{aligned} \]

Definition 3 (Row of a matrix)

Definition 3 (Row of a matrix) The $i$th row of an $m \times n$ matrix $\mathbf{A}$ is the $1 \times n$ row vector of the entries in row $i$:

\[ \mathbf{A}_{i,:} = \begin{bmatrix} a_{i1} & a_{i2} & \cdots & a_{in} \end{bmatrix}. \]

Definition 4 (Column of a matrix)

Definition 4 (Column of a matrix) The $j$th column of an $m \times n$ matrix $\mathbf{A}$ is the column vector of the entries in column $j$:

\[ \mathbf{A}_{:,j} = \begin{bmatrix} a_{1j} \\ a_{2j} \\ \vdots \\ a_{mj} \end{bmatrix}. \]

Example 2 (Rows and columns of a matrix)

Example 2 (Rows and columns of a matrix) For $\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}$, the rows are $\mathbf{A}_{1,:} = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix}$ and $\mathbf{A}_{2,:} = \begin{bmatrix} 4 & 5 & 6 \end{bmatrix}$, and the columns are $\mathbf{A}_{:,1} = \begin{bmatrix} 1 \\ 4 \end{bmatrix}$, $\mathbf{A}_{:,2} = \begin{bmatrix} 2 \\ 5 \end{bmatrix}$ and $\mathbf{A}_{:,3} = \begin{bmatrix} 3 \\ 6 \end{bmatrix}$.

Exercise 2 (Reading off a column of a matrix)

Exercise 2 (Reading off a column of a matrix) When each row of a matrix records one observation, each column holds one measured quantity for every observation, so reading off a column extracts that quantity’s values. Let

\[ \mathbf{A} = \begin{bmatrix} 2 & 0 & -1 \\ 5 & 3 & 4 \end{bmatrix}. \]

Find $\mathbf{A}_{:,2}$, the second column of $\mathbf{A}$.

Hint

The $j$th column of an $m \times n$ matrix lists the entries $a_{1j}, \ldots, a_{mj}$ from top to bottom, as in Example 2.

Solution 2

Solution 2. $\mathbf{A}$ has $m = 2$ rows, so its second column holds the two entries $a_{12}$ and $a_{22}$:

\[ \mathbf{A}_{:,2} = \begin{bmatrix} a_{12} \\ a_{22} \end{bmatrix} = \begin{bmatrix} 0 \\ 3 \end{bmatrix}. \]

Exercise 3 (Counting the rows and columns of a matrix)

Exercise 3 (Counting the rows and columns of a matrix) Data with one row per observation and one column per measured quantity is stored as a matrix, so the matrix’s number of rows is the number of observations and its number of columns is the number of quantities. Let

\[ \mathbf{C} = \begin{bmatrix} 4 & 0 \\ 1 & -3 \\ 2 & 7 \end{bmatrix}. \]

Find $m$ and $n$ such that $\mathbf{C}$ is an $m \times n$ matrix.

Hint

In Definition 1, $m$ counts the rows and $n$ counts the columns.

Solution 3

Solution 3. $\mathbf{C}$ has three rows, $\begin{bmatrix} 4 & 0 \end{bmatrix}$, $\begin{bmatrix} 1 & -3 \end{bmatrix}$ and $\begin{bmatrix} 2 & 7 \end{bmatrix}$, and each row has two entries, so $\mathbf{C}$ has two columns. So $m = 3$ and $n = 2$: $\mathbf{C}$ is a $3 \times 2$ matrix, with $3 \cdot 2 = 6$ entries.

Exercise 4 (Reading off a row of a matrix)

Exercise 4 (Reading off a row of a matrix) When each row of a matrix records one observation, reading off a row extracts every measured quantity for that one observation. Let

\[ \mathbf{D} = \begin{bmatrix} 1 & -4 & 2 \\ 0 & 6 & 3 \\ 5 & 5 & -1 \end{bmatrix}. \]

Find $\mathbf{D}_{3,:}$, the third row of $\mathbf{D}$.

Hint

A row of a matrix lists the entries $a_{i1}, \ldots, a_{in}$ from left to right, as in Example 2.

Solution 4

Solution 4. Write $d_{ij}$ for the entry of $\mathbf{D}$ in row $i$ and column $j$. $\mathbf{D}$ has $n = 3$ columns, so its third row holds the three entries $d_{31}$, $d_{32}$ and $d_{33}$:

\[ \mathbf{D}_{3,:} = \begin{bmatrix} d_{31} & d_{32} & d_{33} \end{bmatrix} = \begin{bmatrix} 5 & 5 & -1 \end{bmatrix}. \]

Theorem 1 (A matrix is a column of its rows)

Theorem 1 (A matrix is a column of its rows) An $m \times n$ matrix $\mathbf{A}$ is the column vector whose entries are its rows:

\[ \mathbf{A} = \begin{bmatrix} \mathbf{A}_{1,:} \\ \vdots \\ \mathbf{A}_{m,:} \end{bmatrix}. \]

Exercise 5 (Stacking rows into a matrix)

Exercise 5 (Stacking rows into a matrix) Theorem 1 says that a matrix is determined by its rows, so data recorded one observation at a time can be assembled into a matrix by stacking those records. Let $\mathbf{M}$ be the $3 \times 2$ matrix with

\[ \mathbf{M}_{1,:} = \begin{bmatrix} 1 & 2 \end{bmatrix}, \qquad \mathbf{M}_{2,:} = \begin{bmatrix} 3 & -1 \end{bmatrix}, \qquad \mathbf{M}_{3,:} = \begin{bmatrix} 0 & 4 \end{bmatrix}. \]

What is $\mathbf{M}_{:,2}$, the second column of $\mathbf{M}$?

Hint

Stack the rows from top to bottom, as in Theorem 1, then read down the second column.

Solution 5

Solution 5. Stacking the three rows from top to bottom gives

\[ \begin{aligned} \mathbf{M} &= \begin{bmatrix} \mathbf{M}_{1,:} \\ \mathbf{M}_{2,:} \\ \mathbf{M}_{3,:} \end{bmatrix} && \text{(}\href{#thm-matrix-column-of-rows}{\text{Theorem~1}}\text{)} \\ &= \begin{bmatrix} 1 & 2 \\ 3 & -1 \\ 0 & 4 \end{bmatrix}, && \text{(substitute the rows)} \end{aligned} \]

so the second column of $\mathbf{M}$ lists the second entry of each row:

\[ \mathbf{M}_{:,2} = \begin{bmatrix} 2 \\ -1 \\ 4 \end{bmatrix}. \]

Theorem 2 (A matrix is a row of its columns)

Theorem 2 (A matrix is a row of its columns) An $m \times n$ matrix $\mathbf{A}$ is the row vector whose entries are its columns:

\[ \mathbf{A} = \begin{bmatrix} \mathbf{A}_{:,1} & \cdots & \mathbf{A}_{:,n} \end{bmatrix}. \]

Exercise 6 (Placing columns side by side)

Exercise 6 (Placing columns side by side) Theorem 2 says that a matrix is determined by its columns, so measurements recorded one quantity at a time can be assembled into a matrix by placing those lists side by side. Let $\mathbf{N}$ be the $2 \times 3$ matrix with

\[ \mathbf{N}_{:,1} = \begin{bmatrix} 2 \\ -1 \end{bmatrix}, \qquad \mathbf{N}_{:,2} = \begin{bmatrix} 0 \\ 3 \end{bmatrix}, \qquad \mathbf{N}_{:,3} = \begin{bmatrix} 4 \\ 1 \end{bmatrix}. \]

What is $\mathbf{N}_{2,:}$, the second row of $\mathbf{N}$?

Hint

Place the columns from left to right, as in Theorem 2, then read across the second row.

Solution 6

Solution 6. Placing the three columns from left to right gives

\[ \begin{aligned} \mathbf{N} &= \begin{bmatrix} \mathbf{N}_{:,1} & \mathbf{N}_{:,2} & \mathbf{N}_{:,3} \end{bmatrix} && \text{(}\href{#thm-matrix-row-of-columns}{\text{Theorem~2}}\text{)} \\ &= \begin{bmatrix} 2 & 0 & 4 \\ -1 & 3 & 1 \end{bmatrix}, && \text{(substitute the columns)} \end{aligned} \]

so the second row of $\mathbf{N}$ lists the second entry of each column:

\[ \mathbf{N}_{2,:} = \begin{bmatrix} -1 & 3 & 1 \end{bmatrix}. \]

Definition 5 (Tensor)

Definition 5 (Tensor) A tensor [2] of order $k \ge 1$ is an array of numbers with $k$ indices, \[ \mathcal{A} = \mathopen{}\left(a_{i_1 i_2 \cdots i_k}\right)\mathclose{}, \quad i_1 = 1, \ldots, n_1, \ \ldots, \ i_k = 1, \ldots, n_k, \] and its dimensions are $n_1 \times n_2 \times \cdots \times n_k$.

Example 3 (Vectors, matrices and images as tensors)

Example 3 (Vectors, matrices and images as tensors) A vector of length $p$ is a tensor of order $1$, and an $m \times n$ matrix is a tensor of order $2$. A $28 \times 28$ grayscale image is a matrix: the entry $a_{ij}$ is the brightness of the pixel in row $i$ and column $j$. A $28 \times 28$ color image stores three numbers per pixel (red, green, blue), so it is an order-3 tensor with dimensions $28 \times 28 \times 3$, and $a_{ijc}$ is the value of color channel $c$ at the pixel in row $i$ and column $j$. It has $28 \cdot 28 \cdot 3 = 2352$ entries.

Exercise 7 (Counting the entries of a batch of photos)

Exercise 7 (Counting the entries of a batch of photos) Software for neural networks often stores a batch of color photos as one tensor, with one index for the photo, two for the pixel’s position, and one for the color channel, and the software stores one number per entry, so the count of entries determines how much memory the batch needs. Let $\mathcal{B}$ be the order-4 tensor for a batch of $10$ color photos of $32 \times 32$ pixels, with $n_1 = 10$, $n_2 = 32$, $n_3 = 32$ and $n_4 = 3$.

How many entries does $\mathcal{B}$ have?

Hint

Each index $i_r$ in Definition 5 takes $n_r$ values, and each choice of all four indices picks one entry, as in Example 3.

Solution 7

Solution 7. An entry of $\mathcal{B}$ is picked by choosing $i_1$ from $10$ values, $i_2$ and $i_3$ from $32$ values each, and $i_4$ from $3$ values, so the number of entries is

\[ \begin{aligned} n_1 \cdot n_2 \cdot n_3 \cdot n_4 &= 10 \cdot 32 \cdot 32 \cdot 3 && \text{(substitute the index ranges)} \\ &= 30720. && \text{(multiply)} \end{aligned} \]

1 Matrix transpose

Definition 6 (Matrix transpose)

Definition 6 (Matrix transpose) The transpose [3] of an $m \times n$ matrix $\mathbf{A}$ is the $n \times m$ matrix ${\mathbf{A}}^{\top}$ obtained by swapping the rows and columns of $\mathbf{A}$:

\[({\mathbf{A}}^{\top})_{ij} = a_{ji}\]

Example 4 (Transposing a $2 \times 3$ matrix)

Example 4 (Transposing a $2 \times 3$ matrix) For $\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}$,

\[ {\mathbf{A}}^{\top} = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix}, \]

a $3 \times 2$ matrix. For instance, entry $(1, 2)$ of the transpose is $a_{21} = 4$.

Exercise 8 (Transposing a $3 \times 2$ matrix)

Exercise 8 (Transposing a $3 \times 2$ matrix) The transpose turns the rows of a matrix into columns, just as the transpose of a vector turns a column into a row. Let

\[ \mathbf{B} = \begin{bmatrix} 1 & -2 \\ 0 & 3 \\ 4 & 5 \end{bmatrix}. \]

Compute ${\mathbf{B}}^{\top}$.

Hint

Entry $(i, j)$ of the transpose is entry $(j, i)$ of the original matrix, as in Example 4.

Solution 8

Solution 8. $\mathbf{B}$ is $3 \times 2$, so its transpose is $2 \times 3$. The $i$th row of ${\mathbf{B}}^{\top}$ is the $i$th column of $\mathbf{B}$:

\[ {\mathbf{B}}^{\top} = \begin{bmatrix} 1 & 0 & 4 \\ -2 & 3 & 5 \end{bmatrix}. \]

For instance, entry $(2, 3)$ of ${\mathbf{B}}^{\top}$ is $b_{32} = 5$.

2 Matrix addition

Definition 7 (Zero matrix)

Definition 7 (Zero matrix) The $m \times n$ zero matrix [4] $\mathbf{0}_{m \times n}$ (or $\mathbf{0}$ when dimensions are clear from context) has all entries equal to zero:

\[ \mathbf{0}_{m \times n} = \begin{bmatrix} 0 & 0 & \cdots & 0 \\ 0 & 0 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 0 \end{bmatrix} \]

Example 5 (Zero matrices of two shapes)

Example 5 (Zero matrices of two shapes) \[ \mathbf{0}_{2 \times 3} = \begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix}, \qquad \mathbf{0}_{3 \times 2} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \\ 0 & 0 \end{bmatrix}. \]

Both have all entries $0$, but one is $2 \times 3$ and the other $3 \times 2$, so they are different matrices.

Exercise 9 (Whether a matrix of zeros is a given zero matrix)

Exercise 9 (Whether a matrix of zeros is a given zero matrix) A zero matrix carries its shape in its subscript, because, as Example 5 shows, zero matrices of different shapes are different matrices. Let

\[ \mathbf{Z} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \\ 0 & 0 \end{bmatrix}. \]

Is $\mathbf{Z} = \mathbf{0}_{2 \times 3}$?

Hint

Compare the shapes of the two matrices, using Definition 2.

Solution 9

Solution 9. No. $\mathbf{Z}$ has three rows and two columns, so it is $3 \times 2$. But $\mathbf{0}_{2 \times 3}$ is $2 \times 3$. Two matrices of different shapes are not equal, even when every entry of both is $0$. Instead, $\mathbf{Z}$ is the zero matrix $\mathbf{0}_{3 \times 2}$.

Definition 8 (Matrix addition)

Definition 8 (Matrix addition) Two matrices $\mathbf{A}$ and $\mathbf{B}$ of the same dimensions $m \times n$ can be added element-wise; their matrix sum [5] is:

\[(\mathbf{A} + \mathbf{B})_{ij} = a_{ij} + b_{ij}\]

Example 6 (Adding matrices, and a sum that is not defined)

Example 6 (Adding matrices, and a sum that is not defined) \[ \begin{aligned} \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} &= \begin{bmatrix} 1 + 0 & 2 + 1 \\ 3 + 1 & 4 + 0 \end{bmatrix} && \text{(add entry by entry)} \\ &= \begin{bmatrix} 1 & 3 \\ 4 & 4 \end{bmatrix}. && \text{(add)} \end{aligned} \]

A $2 \times 2$ matrix and a $2 \times 3$ matrix cannot be added: they do not have the same dimensions, so entry $(1, 3)$ of the sum would have no first term.

Exercise 10 (Adding two $2 \times 2$ matrices)

Exercise 10 (Adding two $2 \times 2$ matrices) A matrix sum combines two tables of the same shape entry by entry, such as two months of counts recorded in the same rows and columns. Let

\[ \mathbf{A} = \begin{bmatrix} 1 & -1 \\ 2 & 0 \end{bmatrix}, \qquad \mathbf{B} = \begin{bmatrix} 3 & 2 \\ -2 & 5 \end{bmatrix}. \]

Compute $\mathbf{A} + \mathbf{B}$.

Hint

Add the entries in the same position, as in Example 6.

Solution 10

Solution 10. Both matrices are $2 \times 2$, so the matrix sum is defined:

\[ \begin{aligned} \mathbf{A} + \mathbf{B} &= \begin{bmatrix} 1 + 3 & -1 + 2 \\ 2 + (-2) & 0 + 5 \end{bmatrix} && \text{(add entry by entry)} \\ &= \begin{bmatrix} 4 & 1 \\ 0 & 5 \end{bmatrix}. && \text{(add)} \end{aligned} \]

Theorem 3 (Matrix addition is commutative)

Theorem 3 (Matrix addition is commutative) For $m \times n$ matrices $\mathbf{A}$ and $\mathbf{B}$, matrix addition is commutative:

\[\mathbf{A} + \mathbf{B} = \mathbf{B} + \mathbf{A}\]

Exercise 11 (Proving that matrix addition is commutative)

Exercise 11 (Proving that matrix addition is commutative) Theorem 3 lets two matrices be added in either order, just as two numbers can be (sums are symmetric).

Prove Theorem 3.

Hint

Two $m \times n$ matrices are equal when their $(i, j)$ entries agree for every $i$ and $j$. Express the $(i, j)$ entry of each side with Definition 8.

Solution 11

Solution 11. Both sides are $m \times n$ matrices, so they are equal when their $(i, j)$ entries agree for every $i$ and $j$. For each $i$ and $j$,

\[ \begin{aligned} (\mathbf{A} + \mathbf{B})_{ij} &= a_{ij} + b_{ij} && \text{(}\href{#def-matrix-addition}{\text{Definition~8}}\text{)} \\ &= b_{ij} + a_{ij} && \text{(sums of numbers are symmetric)} \\ &= (\mathbf{B} + \mathbf{A})_{ij}, && \text{(}\href{#def-matrix-addition}{\text{Definition~8}}\text{)} \end{aligned} \]

where the middle step uses the symmetry of sums of numbers. So $\mathbf{A} + \mathbf{B} = \mathbf{B} + \mathbf{A}$.

Theorem 4 (Matrix addition is associative)

Theorem 4 (Matrix addition is associative) For $m \times n$ matrices $\mathbf{A}$, $\mathbf{B}$, and $\mathbf{C}$, matrix addition is associative:

\[(\mathbf{A} + \mathbf{B}) + \mathbf{C} = \mathbf{A} + (\mathbf{B} + \mathbf{C})\]

Exercise 12 (Proving that matrix addition is associative)

Exercise 12 (Proving that matrix addition is associative) Theorem 4 lets three matrices be added without parentheses, just as three numbers can be (sums are associative).

Prove Theorem 4.

Hint

Two $m \times n$ matrices are equal when their $(i, j)$ entries agree for every $i$ and $j$. Apply Definition 8 twice on each side.

Solution 12

Solution 12. Both sides are $m \times n$ matrices, so they are equal when their $(i, j)$ entries agree for every $i$ and $j$. For each $i$ and $j$,

\[ \begin{aligned} ((\mathbf{A} + \mathbf{B}) + \mathbf{C})_{ij} &= (\mathbf{A} + \mathbf{B})_{ij} + c_{ij} && \text{(}\href{#def-matrix-addition}{\text{Definition~8}}\text{)} \\ &= (a_{ij} + b_{ij}) + c_{ij} && \text{(}\href{#def-matrix-addition}{\text{Definition~8}}\text{)} \\ &= a_{ij} + (b_{ij} + c_{ij}) && \text{(sums of numbers are associative)} \\ &= a_{ij} + (\mathbf{B} + \mathbf{C})_{ij} && \text{(}\href{#def-matrix-addition}{\text{Definition~8}}\text{)} \\ &= (\mathbf{A} + (\mathbf{B} + \mathbf{C}))_{ij}, && \text{(}\href{#def-matrix-addition}{\text{Definition~8}}\text{)} \end{aligned} \]

where the middle step uses the associativity of sums of numbers. So $(\mathbf{A} + \mathbf{B}) + \mathbf{C} = \mathbf{A} + (\mathbf{B} + \mathbf{C})$.

Theorem 5 (Zero matrix is the additive identity)

Theorem 5 (Zero matrix is the additive identity) For an $m \times n$ matrix $\mathbf{A}$:

\[\mathbf{A} + \mathbf{0}_{m \times n} = \mathbf{A}\]

Exercise 13 (Proving that the zero matrix is the additive identity)

Exercise 13 (Proving that the zero matrix is the additive identity) Theorem 5 says that adding the zero matrix leaves a matrix unchanged, just as adding the number $0$ leaves a number unchanged.

Prove Theorem 5.

Hint

Express the $(i, j)$ entry of $\mathbf{A} + \mathbf{0}_{m \times n}$ with Definition 8, and use that every entry of $\mathbf{0}_{m \times n}$ is $0$.

Solution 13

Solution 13. Both sides are $m \times n$ matrices, so they are equal when their $(i, j)$ entries agree for every $i$ and $j$. For each $i$ and $j$,

\[ \begin{aligned} (\mathbf{A} + \mathbf{0}_{m \times n})_{ij} &= a_{ij} + (\mathbf{0}_{m \times n})_{ij} && \text{(}\href{#def-matrix-addition}{\text{Definition~8}}\text{)} \\ &= a_{ij} + 0 && \text{(}\href{#def-zero-matrix}{\text{Definition~7}}\text{)} \\ &= a_{ij}. && \text{(adding } 0 \text{ to a number leaves it unchanged)} \end{aligned} \]

So $\mathbf{A} + \mathbf{0}_{m \times n} = \mathbf{A}$.

Theorem 6 (Additive inverse)

Theorem 6 (Additive inverse) For any $m \times n$ matrix $\mathbf{A}$, the $m \times n$ matrix $-\mathbf{A}$ (defined by $(-\mathbf{A})_{ij} = -a_{ij}$) satisfies:

\[\mathbf{A} + (-\mathbf{A}) = \mathbf{0}_{m \times n}\]

Exercise 14 (Proving that a matrix plus its negation is zero)

Exercise 14 (Proving that a matrix plus its negation is zero) Theorem 6 says that adding $-\mathbf{A}$ to a matrix $\mathbf{A}$ gives the zero matrix, just as adding $-a$ to a number $a$ gives $0$. Here $-\mathbf{A}$, the negation of $\mathbf{A}$, negates every entry of $\mathbf{A}$.

Prove Theorem 6.

Hint

Express the $(i, j)$ entry of $\mathbf{A} + (-\mathbf{A})$ with Definition 8, and use $(-\mathbf{A})_{ij} = -a_{ij}$ from Theorem 6.

Solution 14

Solution 14. Both sides are $m \times n$ matrices, so they are equal when their $(i, j)$ entries agree for every $i$ and $j$. For each $i$ and $j$,

\[ \begin{aligned} (\mathbf{A} + (-\mathbf{A}))_{ij} &= a_{ij} + (-\mathbf{A})_{ij} && \text{(}\href{#def-matrix-addition}{\text{Definition~8}}\text{)} \\ &= a_{ij} + (-a_{ij}) && \text{(definition of } -\mathbf{A} \text{ in }\href{#thm-matadd-inverse}{\text{Theorem~6}}\text{)} \\ &= 0 && \text{(a number plus its negative is } 0 \text{)} \\ &= (\mathbf{0}_{m \times n})_{ij}. && \text{(}\href{#def-zero-matrix}{\text{Definition~7}}\text{)} \end{aligned} \]

So $\mathbf{A} + (-\mathbf{A}) = \mathbf{0}_{m \times n}$.

3 Scalar multiplication

Definition 9 (Scalar multiplication)

Definition 9 (Scalar multiplication) The scalar multiple [6] of a matrix $\mathbf{A}$ by a scalar $c$ is:

\[(c\mathbf{A})_{ij} = c \cdot a_{ij}\]

Example 7 (A scalar multiple)

Example 7 (A scalar multiple) \[ \begin{aligned} 3 \begin{bmatrix} 1 & -2 \\ 0 & 4 \end{bmatrix} &= \begin{bmatrix} 3 \cdot 1 & 3 \cdot(-2) \\ 3 \cdot 0 & 3 \cdot 4 \end{bmatrix} \\ &= \begin{bmatrix} 3 & -6 \\ 0 & 12 \end{bmatrix}. \end{aligned} \]

Exercise 15 (Multiplying a matrix by a negative factor)

Exercise 15 (Multiplying a matrix by a negative factor) A scalar multiple rescales every entry of a matrix by the same factor, as when a table of distances is converted from meters to centimeters. A negative factor flips the sign of every entry, and a factor between $-1$ and $1$ shrinks every entry toward $0$. Let

\[ \mathbf{C} = \begin{bmatrix} 4 & -2 \\ 0 & 6 \end{bmatrix}. \]

Compute $-\frac{1}{2} \mathbf{C}$.

Hint

Multiply each entry by $-\frac{1}{2}$, as in Example 7.

Solution 15

Solution 15. \[ \begin{aligned} -\frac{1}{2} \mathbf{C} &= \begin{bmatrix} -\frac{1}{2} \cdot 4 & -\frac{1}{2} \cdot(-2) \\ -\frac{1}{2} \cdot 0 & -\frac{1}{2} \cdot 6 \end{bmatrix} && \text{(}\href{#def-scalar-mult}{\text{Definition~9}}\text{)} \\ &= \begin{bmatrix} -2 & 1 \\ 0 & -3 \end{bmatrix}. && \text{(multiply)} \end{aligned} \]

4 Matrix multiplication

Definition 10 (Matrix multiplication)

Definition 10 (Matrix multiplication) The product [7] of an $m \times k$ matrix $\mathbf{A}$ and a $k \times n$ matrix $\mathbf{B}$ is the $m \times n$ matrix $\mathbf{C} = \mathbf{A}\mathbf{B}$ with entries:

\[c_{ij} = \sum_{s=1}^{k} a_{is}\, b_{sj}\]

Remark 2 (Matrix multiplication is not commutative)

Remark 2 (Matrix multiplication is not commutative). When $\mathbf{A}\mathbf{B}$ and $\mathbf{B}\mathbf{A}$ are both defined, they are usually different. For example, with

\[ \mathbf{A} = \begin{bmatrix} 1 & 2 \\ 0 & 1 \end{bmatrix}, \qquad \mathbf{B} = \begin{bmatrix} 1 & 0 \\ 1 & 1 \end{bmatrix}, \]

\[ \begin{aligned} \mathbf{A}\mathbf{B} &= \begin{bmatrix} 3 & 2 \\ 1 & 1 \end{bmatrix} \\ \mathbf{B}\mathbf{A} &= \begin{bmatrix} 1 & 2 \\ 1 & 3 \end{bmatrix}, \end{aligned} \]

so $\mathbf{A}\mathbf{B} \neq \mathbf{B}\mathbf{A}$.

Example 8 (Dot product as matrix multiplication)

Example 8 (Dot product as matrix multiplication) The dot product of two column vectors $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ and $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}$ can be written as a matrix product of the row vector ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}$ with the column vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}$:

\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}} &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}} \\ &= [x_1,\; x_2,\; \ldots,\; x_p] \begin{bmatrix} \beta_{1} \\ \beta_{2} \\ \vdots \\ \beta_{p} \end{bmatrix} \\ &= x_1\beta_{1} + x_2\beta_{2} + \cdots + x_p \beta_{p} \end{aligned} \]

Exercise 16 (Computing the product of a $2 \times 3$ and a $3 \times 2$ matrix)

Exercise 16 (Computing the product of a $2 \times 3$ and a $3 \times 2$ matrix) The product of two matrices is defined only when the first has as many columns as the second has rows. Each entry of the product combines one row of the first factor with one column of the second. Let

\[ \mathbf{A} = \begin{bmatrix} 1 & 0 & 2 \\ -1 & 3 & 1 \end{bmatrix}, \qquad \mathbf{B} = \begin{bmatrix} 2 & 1 \\ 0 & -1 \\ 1 & 4 \end{bmatrix}. \]

Find $\mathbf{A}\mathbf{B}$.

Hint

$\mathbf{A}$ is $2 \times 3$, and $\mathbf{B}$ is $3 \times 2$. By Definition 10, $\mathbf{A}\mathbf{B}$ is therefore $2 \times 2$. Its entry $c_{ij}$ is $a_{i1} b_{1j} + a_{i2} b_{2j} + a_{i3} b_{3j}$.

Solution 16

Solution 16. $\mathbf{A}$ has three columns, and $\mathbf{B}$ has three rows. So the product is defined and is $2 \times 2$. Its entries are

\[ \begin{aligned} \mathbf{A}\mathbf{B} &= \begin{bmatrix} 1 \cdot 2 + 0 \cdot 0 + 2 \cdot 1 & 1 \cdot 1 + 0 \cdot(-1) + 2 \cdot 4 \\ (-1) \cdot 2 + 3 \cdot 0 + 1 \cdot 1 & (-1) \cdot 1 + 3 \cdot(-1) + 1 \cdot 4 \end{bmatrix} && \text{(}\href{#def-matrix-mult}{\text{Definition~10}}\text{)} \\ &= \begin{bmatrix} 4 & 9 \\ -1 & 0 \end{bmatrix}. && \text{(multiply and add)} \end{aligned} \]

Theorem 7 (Matrix multiplication is associative)

Theorem 7 (Matrix multiplication is associative) For an $m \times k$ matrix $\mathbf{A}$, a $k \times l$ matrix $\mathbf{B}$, and an $l \times n$ matrix $\mathbf{C}$:

\[ \underbrace{(\mathbf{A}\mathbf{B})\mathbf{C}}_{m \times n} = \underbrace{\mathbf{A}(\mathbf{B}\mathbf{C})}_{m \times n} \]

Proof

Proof. Entry $(i, j)$ of each side, from Definition 10:

\[ \begin{aligned} \mathopen{}\left[(\mathbf{A}\mathbf{B})\mathbf{C}\right]\mathclose{}_{ij} &= \sum_{t=1}^{l} (\mathbf{A}\mathbf{B})_{it}\, c_{tj} && \text{(definition of } (\mathbf{A}\mathbf{B})\mathbf{C} \text{)} \\ &= \sum_{t=1}^{l} \mathopen{}\left(\sum_{s=1}^{k} a_{is}\, b_{st}\right)\mathclose{} c_{tj} && \text{(definition of } \mathbf{A}\mathbf{B} \text{)} \\ &= \sum_{s=1}^{k} a_{is} \mathopen{}\left(\sum_{t=1}^{l} b_{st}\, c_{tj}\right)\mathclose{} && \text{(distribute and swap the finite sums)} \\ &= \sum_{s=1}^{k} a_{is}\, (\mathbf{B}\mathbf{C})_{sj} && \text{(definition of } \mathbf{B}\mathbf{C} \text{)} \\ &= \mathopen{}\left[\mathbf{A}(\mathbf{B}\mathbf{C})\right]\mathclose{}_{ij} && \text{(definition of } \mathbf{A}(\mathbf{B}\mathbf{C}) \text{)} \end{aligned} \]

Exercise 17 (Grouping a product of three matrices)

Exercise 17 (Grouping a product of three matrices) By Theorem 7, a product of three matrices can be grouped either way, so $\mathbf{A}\mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ can be computed as $\mathbf{A}(\mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})$, without ever forming $\mathbf{A}\mathbf{B}$. Let

\[ \mathbf{A} = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}, \qquad \mathbf{B} = \begin{bmatrix} 2 & 0 \\ 1 & 1 \end{bmatrix}, \qquad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \begin{bmatrix} 1 \\ 2 \end{bmatrix}, \]

where $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ is a $2 \times 1$ matrix.

Find $\mathbf{A}\mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ by computing $\mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ first.

Hint

By Definition 10, $\mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ is a $2 \times 1$ matrix; multiply $\mathbf{A}$ by it.

Solution 17

Solution 17. First,

\[ \begin{aligned} \mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= \begin{bmatrix} 2 \cdot 1 + 0 \cdot 2 \\ 1 \cdot 1 + 1 \cdot 2 \end{bmatrix} && \text{(}\href{#def-matrix-mult}{\text{Definition~10}}\text{)} \\ &= \begin{bmatrix} 2 \\ 3 \end{bmatrix}, && \text{(multiply and add)} \end{aligned} \]

\[ \begin{aligned} \mathbf{A}\mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= \mathbf{A}(\mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) && \text{(}\href{#thm-matmul-assoc}{\text{Theorem~7}}\text{)} \\ &= \begin{bmatrix} 1 \cdot 2 + 1 \cdot 3 \\ 0 \cdot 2 + 1 \cdot 3 \end{bmatrix} && \text{(}\href{#def-matrix-mult}{\text{Definition~10}}\text{)} \\ &= \begin{bmatrix} 5 \\ 3 \end{bmatrix}. && \text{(multiply and add)} \end{aligned} \]

Grouping the other way gives the same answer: $\mathbf{A}\mathbf{B} = \begin{bmatrix} 3 & 1 \\ 1 & 1 \end{bmatrix}$, and $(\mathbf{A}\mathbf{B})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \begin{bmatrix} 3 + 2 \\ 1 + 2 \end{bmatrix} = \begin{bmatrix} 5 \\ 3 \end{bmatrix}$.

Theorem 8 (Matrix multiplication is distributive over addition)

Theorem 8 (Matrix multiplication is distributive over addition) Matrix multiplication distributes over addition from the left and from the right. For an $m \times k$ matrix $\mathbf{A}$ and $k \times n$ matrices $\mathbf{B}$ and $\mathbf{C}$:

\[\mathbf{A}(\mathbf{B} + \mathbf{C}) = \mathbf{A}\mathbf{B} + \mathbf{A}\mathbf{C}\]

For $m \times k$ matrices $\mathbf{A}$ and $\mathbf{B}$ and a $k \times n$ matrix $\mathbf{C}$:

\[(\mathbf{A} + \mathbf{B})\mathbf{C} = \mathbf{A}\mathbf{C} + \mathbf{B}\mathbf{C}\]

Exercise 18 (Proving that matrix multiplication distributes from the left)

Exercise 18 (Proving that matrix multiplication distributes from the left) Theorem 8 lets a product with a matrix sum be expanded term by term, just as multiplication of numbers distributes over addition. Let $\mathbf{A}$ be an $m \times k$ matrix, and let $\mathbf{B}$ and $\mathbf{C}$ be $k \times n$ matrices.

Prove that $\mathbf{A}(\mathbf{B} + \mathbf{C}) = \mathbf{A}\mathbf{B} + \mathbf{A}\mathbf{C}$.

Hint

Express the $(i, j)$ entry of the left side with Definition 10 and Definition 8, then split the result with the rule for a sum of sums.

Solution 18

Solution 18. Both sides are $m \times n$ matrices, so they are equal when their $(i, j)$ entries agree for every $i$ and $j$. For each $i$ and $j$,

\[ \begin{aligned} \mathopen{}\left[\mathbf{A}(\mathbf{B} + \mathbf{C})\right]\mathclose{}_{ij} &= \sum_{s=1}^{k} a_{is} (\mathbf{B} + \mathbf{C})_{sj} && \text{(}\href{#def-matrix-mult}{\text{Definition~10}}\text{)} \\ &= \sum_{s=1}^{k} a_{is} (b_{sj} + c_{sj}) && \text{(}\href{#def-matrix-addition}{\text{Definition~8}}\text{)} \\ &= \sum_{s=1}^{k} \mathopen{}\left(a_{is} b_{sj} + a_{is} c_{sj}\right)\mathclose{} && \text{(multiplication of numbers distributes over addition)} \\ &= \sum_{s=1}^{k} a_{is} b_{sj} + \sum_{s=1}^{k} a_{is} c_{sj} && \text{(a sum of sums splits)} \\ &= (\mathbf{A}\mathbf{B})_{ij} + (\mathbf{A}\mathbf{C})_{ij} && \text{(}\href{#def-matrix-mult}{\text{Definition~10}}\text{)} \\ &= (\mathbf{A}\mathbf{B} + \mathbf{A}\mathbf{C})_{ij}. && \text{(}\href{#def-matrix-addition}{\text{Definition~8}}\text{)} \end{aligned} \]

The third step uses that multiplication of numbers distributes over addition, and the fourth uses the rule for a sum of sums. So $\mathbf{A}(\mathbf{B} + \mathbf{C}) = \mathbf{A}\mathbf{B} + \mathbf{A}\mathbf{C}$.

Theorem 9 (Product of two sums of matrices)

Theorem 9 (Product of two sums of matrices) For $m \times k$ matrices $\mathbf{A}$ and $\mathbf{B}$ and $k \times n$ matrices $\mathbf{C}$ and $\mathbf{D}$,

\[ (\mathbf{A} + \mathbf{B})(\mathbf{C} + \mathbf{D}) = \mathbf{A}\mathbf{C} + \mathbf{A}\mathbf{D} + \mathbf{B}\mathbf{C} + \mathbf{B}\mathbf{D}. \]

Each product keeps its factors in the order they appear on the left, because matrix multiplication is not commutative. So $\mathbf{A}\mathbf{D}$ and $\mathbf{B}\mathbf{C}$ are not collected into $2\mathbf{A}\mathbf{D}$.

Proof

Proof. \[ \begin{aligned} (\mathbf{A} + \mathbf{B})(\mathbf{C} + \mathbf{D}) &= (\mathbf{A} + \mathbf{B})\mathbf{C} + (\mathbf{A} + \mathbf{B})\mathbf{D} && \text{(}\href{#thm-matmul-distrib}{\text{Theorem~8}}\text{, from the left)} \\ &= \mathbf{A}\mathbf{C} + \mathbf{B}\mathbf{C} + \mathbf{A}\mathbf{D} + \mathbf{B}\mathbf{D} && \text{(}\href{#thm-matmul-distrib}{\text{Theorem~8}}\text{, from the right, twice)} \\ &= \mathbf{A}\mathbf{C} + \mathbf{A}\mathbf{D} + \mathbf{B}\mathbf{C} + \mathbf{B}\mathbf{D} && \text{(matrix addition is commutative)} \end{aligned} \]

Example 9 (Why the middle terms do not combine)

Example 9 (Why the middle terms do not combine) Take $\mathbf{A} = \mathbf{C} = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}$ and $\mathbf{B} = \mathbf{D} = \begin{bmatrix} 1 & 0 \\ 1 & 1 \end{bmatrix}$. Then $\mathbf{A} + \mathbf{B} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$, so

\[ \begin{aligned} (\mathbf{A} + \mathbf{B})^2 &= \begin{bmatrix} 5 & 4 \\ 4 & 5 \end{bmatrix}, \\ \mathbf{A}^2 + \mathbf{A}\mathbf{B} + \mathbf{B}\mathbf{A} + \mathbf{B}^2 &= \begin{bmatrix} 1 & 2 \\ 0 & 1 \end{bmatrix} + \begin{bmatrix} 2 & 1 \\ 1 & 1 \end{bmatrix} + \begin{bmatrix} 1 & 1 \\ 1 & 2 \end{bmatrix} + \begin{bmatrix} 1 & 0 \\ 2 & 1 \end{bmatrix} \\ &= \begin{bmatrix} 5 & 4 \\ 4 & 5 \end{bmatrix}. \end{aligned} \]

Here $\mathbf{A}\mathbf{B} \neq \mathbf{B}\mathbf{A}$, so the middle terms do not combine into $2\mathbf{A}\mathbf{B}$. The squared norm of a sum of vectors, $\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rVert\mathclose{}^2$, is a case where they do combine (Theorem 2 in Inner Products and Orthogonality), because $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$. Matrices that commute, such as $\mathbf{A}$ and $\mathbf{A}^2$, combine the same way.

Exercise 19 (Comparing $(\mathbf{A} + \mathbf{B})(\mathbf{A} + \mathbf{B})$ with the rule for numbers)

Exercise 19 (Comparing $(\mathbf{A} + \mathbf{B})(\mathbf{A} + \mathbf{B})$ with the rule for numbers) For numbers, $(a + b)^2 = a^2 + 2ab + b^2$. Theorem 9 expands a product of two matrix sums into four terms instead, because $\mathbf{A}\mathbf{B}$ and $\mathbf{B}\mathbf{A}$ can differ. Let

\[ \mathbf{A} = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}, \qquad \mathbf{B} = \begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix}. \]

Is $(\mathbf{A} + \mathbf{B})(\mathbf{A} + \mathbf{B})$ equal to $\mathbf{A}\mathbf{A} + 2\mathbf{A}\mathbf{B} + \mathbf{B}\mathbf{B}$?

Hint

Compute $\mathbf{A} + \mathbf{B}$ and its product with itself, then compute $\mathbf{A}\mathbf{A}$, $\mathbf{A}\mathbf{B}$ and $\mathbf{B}\mathbf{B}$ with Definition 10.

Solution 19

Solution 19. No. On the left, $\mathbf{A} + \mathbf{B} = \begin{bmatrix} 1 & 1 \\ 0 & 0 \end{bmatrix}$, so

\[ \begin{aligned} (\mathbf{A} + \mathbf{B})(\mathbf{A} + \mathbf{B}) &= \begin{bmatrix} 1 \cdot 1 + 1 \cdot 0 & 1 \cdot 1 + 1 \cdot 0 \\ 0 \cdot 1 + 0 \cdot 0 & 0 \cdot 1 + 0 \cdot 0 \end{bmatrix} && \text{(}\href{#def-matrix-mult}{\text{Definition~10}}\text{)} \\ &= \begin{bmatrix} 1 & 1 \\ 0 & 0 \end{bmatrix}. && \text{(multiply and add)} \end{aligned} \]

On the right, the same rule gives $\mathbf{A}\mathbf{A} = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}$, $\mathbf{A}\mathbf{B} = \begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix}$ and $\mathbf{B}\mathbf{B} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \end{bmatrix}$, so

\[ \mathbf{A}\mathbf{A} + 2\mathbf{A}\mathbf{B} + \mathbf{B}\mathbf{B} = \begin{bmatrix} 1 & 2 \\ 0 & 0 \end{bmatrix}. \]

The entries in position $(1, 2)$ differ, $1 \ne 2$, so the two matrices are not equal. The rule for numbers fails because $\mathbf{B}\mathbf{A} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \end{bmatrix}$ differs from $\mathbf{A}\mathbf{B}$: Theorem 9 gives $\mathbf{A}\mathbf{A} + \mathbf{A}\mathbf{B} + \mathbf{B}\mathbf{A} + \mathbf{B}\mathbf{B} = \begin{bmatrix} 1 & 1 \\ 0 & 0 \end{bmatrix}$, which does match the left side.

Exercise 20 (Which of these products are defined?)

Exercise 20 (Which of these products are defined?) Let $\mathbf{A}$ be $4 \times 3$, let $\mathbf{B}$ be $3 \times 2$, and let $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^3$. For each expression, say whether it is defined, and if so give the shape of the result:

$\mathbf{A}\mathbf{B}$
$\mathbf{B}\mathbf{A}$
$\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$
$\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$
$\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}^{\top}$

Solution 20

Solution 20. A vector in $\mathbb{R}^3$ is a $3 \times 1$ matrix, so the same rule settles every case.

$\mathbf{A}\mathbf{B}$: $(4 \times 3)(3 \times 2)$. The inner pair is $3$ and $3$, so it is defined, and the result is $4 \times 2$.
$\mathbf{B}\mathbf{A}$: $(3 \times 2)(4 \times 3)$. The inner pair is $2$ and $4$, which do not match, so it is not defined. Matrix multiplication is not commutative, and this is the blunt form of that: swapping the order can leave an expression that means nothing.
$\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$: $(4 \times 3)(3 \times 1)$, defined, result $4 \times 1$ — a vector in $\mathbb{R}^4$.
$\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$: $(1 \times 3)(3 \times 1)$, defined, result $1 \times 1$ — a single number. The product is the inner product of Equation 1 in Vectors.
$\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}^{\top}$: $(3 \times 1)(1 \times 3)$, defined, result $3 \times 3$ — a matrix.

The last two use the same two vectors and differ only in order, and they return objects of different kinds. Reading a transpose as decoration rather than as a shape change is the commonest way to lose track of an expression.

5 Square and identity matrices

Definition 11 (Square matrix)

Definition 11 (Square matrix) A matrix is square [8] if it has the same number of rows as columns.

Example 10 (Square and not square)

Example 10 (Square and not square) $\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$ is square ($2$ rows, $2$ columns); $\begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}$ is not ($2$ rows, $3$ columns).

Exercise 21 (Whether a data matrix times its transpose is square)

Exercise 21 (Whether a data matrix times its transpose is square) Many statistical methods start from an $n \times p$ data matrix $\mathbf{X}$, with one row per observation, and form the product ${\mathbf{X}}^{\top}\mathbf{X}$ of its transpose with itself. Whether that product is square matters, because some operations, such as taking the inverse of a matrix, apply only to square matrices.

Is ${\mathbf{X}}^{\top}\mathbf{X}$ square?

Hint

Find the shape of ${\mathbf{X}}^{\top}$ from Definition 6, then the shape of the product from Definition 10.

Solution 21

Solution 21. Yes. The transpose ${\mathbf{X}}^{\top}$ is $p \times n$, so ${\mathbf{X}}^{\top}$ has $n$ columns. $\mathbf{X}$ has $n$ rows, so the product ${\mathbf{X}}^{\top}\mathbf{X}$ is defined and is $p \times p$. It has $p$ rows and $p$ columns, so it is square for every $n$ and $p$.

Definition 12 (Main diagonal)

Definition 12 (Main diagonal) The main diagonal [9] of an $m \times n$ matrix $\mathbf{A}$ is the list of its entries $a_{11}, a_{22}, \ldots, a_{kk}$, where $k$ is the smaller of $m$ and $n$.

Definition 13 (Diagonal entry)

Definition 13 (Diagonal entry) The entries on the main diagonal (Definition 12) of a matrix $\mathbf{A}$ are the diagonal entries of $\mathbf{A}$.

Definition 14 (Off-diagonal entry)

Definition 14 (Off-diagonal entry) In an $m \times n$ matrix $\mathbf{A}$ with main diagonal (Definition 12), every entry $a_{ij}$ with $i \ne j$ is an off-diagonal entry.

Example 11 (Diagonal entries of a square and a non-square matrix)

Example 11 (Diagonal entries of a square and a non-square matrix)

For $\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$, the diagonal entries are $a_{11} = 1$ and $a_{22} = 4$, and the off-diagonal entries are $a_{12} = 2$ and $a_{21} = 3$.
For $\mathbf{B} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}$, $m = 2$ and $n = 3$, so $k = 2$, and the diagonal entries are $b_{11} = 1$ and $b_{22} = 5$; the other four entries, $2$, $3$, $4$ and $6$, are off-diagonal.

Exercise 22 (The main diagonal of a tall matrix)

Exercise 22 (The main diagonal of a tall matrix) The main diagonal of a matrix that is not square stops when it runs out of rows or columns, whichever comes first. A tall matrix, with more rows than columns, runs out of columns first. Let

\[ \mathbf{E} = \begin{bmatrix} 7 & -2 \\ 1 & 3 \\ 0 & 5 \\ 4 & -6 \end{bmatrix}. \]

Find the main diagonal of $\mathbf{E}$.

Hint

Find $m$, $n$ and $k$ in Definition 12.

Solution 22

Solution 22. Write $e_{ij}$ for the entry of $\mathbf{E}$ in row $i$ and column $j$. $\mathbf{E}$ is $4 \times 2$, so $m = 4$, $n = 2$ and $k = 2$. The main diagonal of $\mathbf{E}$ is therefore the list $e_{11}, e_{22}$, that is, $7, 3$. The list stops at $e_{22}$ because $\mathbf{E}$ has no third column. It therefore has no entry $e_{33}$.

Exercise 23 (Adding the diagonal entries of a square matrix)

Exercise 23 (Adding the diagonal entries of a square matrix) The sum of the diagonal entries of a square matrix summarizes that matrix in one number, called its trace. Let

\[ \mathbf{F} = \begin{bmatrix} 2 & 5 & -1 \\ 0 & -3 & 8 \\ 4 & 1 & 6 \end{bmatrix}. \]

Find the sum of the diagonal entries of $\mathbf{F}$.

Hint

Find the main diagonal of $\mathbf{F}$ first.

Solution 23

Solution 23. Write $f_{ij}$ for the entry of $\mathbf{F}$ in row $i$ and column $j$. $\mathbf{F}$ is $3 \times 3$, so its diagonal entries are $f_{11} = 2$, $f_{22} = -3$ and $f_{33} = 6$, and adding them gives

\[ f_{11} + f_{22} + f_{33} = 2 + (-3) + 6 = 5. \]

Exercise 24 (Counting the off-diagonal entries of a square matrix)

Exercise 24 (Counting the off-diagonal entries of a square matrix) In a table of pairwise comparisons among several items, the off-diagonal entries are the comparisons between two different items, so counting them counts the ordered pairs of different items. Let $\mathbf{A}$ be a $5 \times 5$ matrix, so $\mathbf{A}$ is square.

How many off-diagonal entries does $\mathbf{A}$ have?

Hint

Count all the entries, then subtract the diagonal entries.

Solution 24

Solution 24. $\mathbf{A}$ has $5 \cdot 5 = 25$ entries. Its diagonal entries are the $5$ entries $a_{ii}$, $i = 1, \ldots, 5$, and every other entry has $i \ne j$, so $\mathbf{A}$ has $25 - 5 = 20$ off-diagonal entries.

Definition 15 (Identity matrix)

Definition 15 (Identity matrix) The $p \times p$ identity matrix [10] $\mathbf{I}_p$ (or $\mathbf{I}$ when the size is clear from context) has ones on the main diagonal and zeros elsewhere:

\[ (\mathbf{I}_p)_{ij} = \begin{cases} 1 & \text{if } i = j \\ 0 & \text{if } i \neq j \end{cases} \qquad \mathbf{I}_p = \begin{bmatrix} 1 & 0 & \cdots & 0 \\ 0 & 1 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 1 \end{bmatrix} \]

Example 12 (The $3 \times 3$ identity)

Example 12 (The $3 \times 3$ identity) \[ \mathbf{I}_3 = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}, \]

so, for instance, $(\mathbf{I}_3)_{22} = 1$ and $(\mathbf{I}_3)_{23} = 0$.

Exercise 25 (Whether a matrix with the identity pattern is an identity matrix)

Exercise 25 (Whether a matrix with the identity pattern is an identity matrix) An identity matrix has ones on its main diagonal and zeros elsewhere. The matrix below has the same pattern of ones and zeros. Let

\[ \mathbf{J} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix}. \]

Is $\mathbf{J}$ an identity matrix?

Hint

Compare the shape of $\mathbf{J}$ with the shape of $\mathbf{I}_p$ in Definition 15.

Solution 25

Solution 25. No. Every identity matrix $\mathbf{I}_p$ is $p \times p$, so it is square. $\mathbf{J}$ is $2 \times 3$, so it is not square. Therefore $\mathbf{J}$ is not an identity matrix, even though its main diagonal holds ones and its other entries are zeros.

Remark 3 (The identity matrix and the Kronecker delta)

Remark 3 (The identity matrix and the Kronecker delta). The entries of the identity matrix are given by the Kronecker delta: $(\mathbf{I}_p)_{ij} = \delta_{ij}$.

Theorem 10 (Identity matrix is a multiplicative identity)

Theorem 10 (Identity matrix is a multiplicative identity) For any $m \times p$ matrix $\mathbf{A}$:

\[\mathbf{A}\,\mathbf{I}_p = \mathbf{A}\]

\[\mathbf{I}_m\,\mathbf{A} = \mathbf{A}\]

Exercise 26 (Multiplying by an identity matrix)

Exercise 26 (Multiplying by an identity matrix) Theorem 10 says that an identity matrix of the right size leaves a matrix unchanged under matrix multiplication, just as multiplying a number by $1$ leaves it unchanged. Let

\[ \mathbf{A} = \begin{bmatrix} 2 & 0 & -1 \\ 5 & 3 & 4 \end{bmatrix}. \]

Compute $\mathbf{I}_2 \mathbf{A}$ entry by entry, from the definition of matrix multiplication.

Hint

Entry $(i, j)$ of $\mathbf{I}_2 \mathbf{A}$ is the dot product of the $i$th row of $\mathbf{I}_2$ with the $j$th column of $\mathbf{A}$, as in Example 8.

Solution 26

Solution 26. $\mathbf{I}_2$ is $2 \times 2$ and $\mathbf{A}$ is $2 \times 3$, so the product is defined and is $2 \times 3$. The first row of $\mathbf{I}_2$ is $\begin{bmatrix} 1 & 0 \end{bmatrix}$ and the second is $\begin{bmatrix} 0 & 1 \end{bmatrix}$, so by Definition 10

\[ \begin{aligned} \mathbf{I}_2 \mathbf{A} &= \begin{bmatrix} 1 \cdot 2 + 0 \cdot 5 & 1 \cdot 0 + 0 \cdot 3 & 1 \cdot(-1) + 0 \cdot 4 \\ 0 \cdot 2 + 1 \cdot 5 & 0 \cdot 0 + 1 \cdot 3 & 0 \cdot(-1) + 1 \cdot 4 \end{bmatrix} && \text{(}\href{#def-matrix-mult}{\text{Definition~10}}\text{)} \\ &= \begin{bmatrix} 2 & 0 & -1 \\ 5 & 3 & 4 \end{bmatrix} && \text{(multiply and add)} \\ &= \mathbf{A}. \end{aligned} \]

Each row of $\mathbf{I}_2$ has a single $1$, so entry $(i, j)$ of the product picks out entry $(i, j)$ of $\mathbf{A}$, as Theorem 10 says.

6 Matrix-vector multiplication

Definition 16 (Matrix-vector multiplication)

Definition 16 (Matrix-vector multiplication) The matrix-vector product of an $m \times p$ matrix $\mathbf{A}$ and a $p \times 1$ column vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ is the $m \times 1$ column vector $\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ with entries:

\[(\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i = \sum_{j=1}^pa_{ij}\, x_j\]

Remark 4 (Each entry is a dot product)

Remark 4 (Each entry is a dot product). Matrix-vector multiplication extends the dot product (Definition 8 in Vectors): entry $i$ of $\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ is the dot product of row $i$ of $\mathbf{A}$ with $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$. For example,

\[ \begin{aligned} \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} 1 \\ -1 \end{bmatrix} &= \begin{bmatrix} 1 \cdot 1 + 2 \cdot(-1) \\ 3 \cdot 1 + 4 \cdot(-1) \end{bmatrix} \\ &= \begin{bmatrix} -1 \\ -1 \end{bmatrix}. \end{aligned} \]

Exercise 27 (A matrix-vector product with more columns than rows)

Exercise 27 (A matrix-vector product with more columns than rows) A matrix-vector product $\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ computes one weighted sum of the entries of $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ for each row of $\mathbf{A}$, so one multiplication computes several weighted sums at once. The matrix $\mathbf{A}$ below has more columns than rows, so $\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ is shorter than $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$. Let

\[ \mathbf{A} = \begin{bmatrix} 2 & 0 & -1 \\ 5 & 3 & 4 \end{bmatrix}, \qquad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}. \]

Compute $\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$.

Hint

Entry $i$ of $\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ is the dot product of the $i$th row of $\mathbf{A}$ with $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$, as in Remark 4.

Solution 27

Solution 27. $\mathbf{A}$ is $2 \times 3$ and $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ is $3 \times 1$, so the matrix-vector product is defined and is $2 \times 1$:

\[ \begin{aligned} \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= \begin{bmatrix} 2 \cdot 1 + 0 \cdot 2 + (-1) \cdot 3 \\ 5 \cdot 1 + 3 \cdot 2 + 4 \cdot 3 \end{bmatrix} && \text{(}\href{#def-matvec-mult}{\text{Definition~16}}\text{)} \\ &= \begin{bmatrix} 2 + 0 - 3 \\ 5 + 6 + 12 \end{bmatrix} && \text{(multiply)} \\ &= \begin{bmatrix} -1 \\ 23 \end{bmatrix}. && \text{(add)} \end{aligned} \]

Definition 17 (Linear system, system of linear equations)

Definition 17 (Linear system, system of linear equations) A linear system [11] (or system of linear equations) with $m$ equations in $n$ unknowns is an equation $\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ in which the $m \times n$ matrix $\mathbf{A}$ and the vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \in \mathbb{R}^m$ are given. A solution of the system is a vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^n$ for which $\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ holds.

Example 13 (Two equations in two unknowns)

Example 13 (Two equations in two unknowns) By Definition 16, row $i$ of $\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ is the equation $a_{i1} x_1 + \cdots + a_{in} x_n = b_i$. So the two equations $x_1 + 2 x_2 = 5$ and $3 x_1 + 4 x_2 = 6$ form the linear system $\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \begin{bmatrix} 5 \\ 6 \end{bmatrix}$. The vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (-4, 4.5)$ is a solution:

\[ \begin{aligned} \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} -4 \\ 4.5 \end{bmatrix} &= \begin{bmatrix} 1 \cdot(-4) + 2 \cdot 4.5 \\ 3 \cdot(-4) + 4 \cdot 4.5 \end{bmatrix} && \text{(}\href{#def-matvec-mult}{\text{Definition~16}}\text{)} \\ &= \begin{bmatrix} -4 + 9 \\ -12 + 18 \end{bmatrix} && \text{(multiply)} \\ &= \begin{bmatrix} 5 \\ 6 \end{bmatrix} && \text{(add)} \end{aligned} \]

The vector $(1, 2)$ is not a solution: its first equation holds, since $1 + 2 \cdot 2 = 5$, but its second gives $3 \cdot 1 + 4 \cdot 2 = 11 \ne 6$.

Exercise 28 (Checking whether a vector solves a linear system)

Exercise 28 (Checking whether a vector solves a linear system) Checking whether a vector solves a linear system takes only one matrix-vector product, which is much less work than solving the system. Let

\[ \mathbf{A} = \begin{bmatrix} 3 & 1 \\ 1 & -2 \end{bmatrix}, \qquad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \begin{bmatrix} 5 \\ 4 \end{bmatrix}, \qquad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \begin{bmatrix} 2 \\ -1 \end{bmatrix}. \]

Is $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ a solution of $\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$?

Hint

Compute $\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ with Definition 16. Then compare it with $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$, as in Example 13.

Solution 28

Solution 28. Yes. The matrix-vector product is

\[ \begin{aligned} \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= \begin{bmatrix} 3 \cdot 2 + 1 \cdot(-1) \\ 1 \cdot 2 + (-2) \cdot(-1) \end{bmatrix} && \text{(}\href{#def-matvec-mult}{\text{Definition~16}}\text{)} \\ &= \begin{bmatrix} 5 \\ 4 \end{bmatrix}, && \text{(multiply and add)} \end{aligned} \]

which matches $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ entry by entry, so $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ is a solution of the system.

7 Linear and affine maps

Definition 18 (Additive function)

Definition 18 (Additive function) A function $f: \mathbb{R}^p \to \mathbb{R}^m$ is additive [12] if it preserves vector addition: for all $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^p$,

\[f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) + f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})\]

Example 14 (An additive function, and one that is not)

Example 14 (An additive function, and one that is not)

$f(x) = 2x$ on $\mathbb{R}$ is additive. For $x, y \in \mathbb{R}$:

\[ \begin{aligned} f(x + y) &= 2(x + y) \\ &= 2x + 2y \\ &= f(x) + f(y) \end{aligned} \]
$g(x) = x + 1$ on $\mathbb{R}$ is not additive, since $g(0 + 0) = 1$ but $g(0) + g(0) = 2$.

Exercise 29 (Testing whether $x_1 x_2$ is additive)

Exercise 29 (Testing whether $x_1 x_2$ is additive) An additive function can be evaluated at $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}$ by evaluating it at $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ and at $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}$ separately, so additivity is worth checking before splitting a computation that way. Let $f: \mathbb{R}^2 \to \mathbb{R}$ be $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = x_1 x_2$.

Is $f$ additive?

Hint

Try $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 0)$ and $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= (0, 1)$ in Definition 18.

Solution 29

Solution 29. No. For $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 0)$ and $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= (0, 1)$, $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= (1, 1)$, so

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) = 1 \cdot 1 = 1, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) + f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) = 1 \cdot 0 + 0 \cdot 1 = 0. \]

These differ, so $f$ does not satisfy Definition 18 for this pair of inputs, and $f$ is not additive.

Definition 19 (Homogeneous function)

Definition 19 (Homogeneous function) A function $f: \mathbb{R}^p \to \mathbb{R}^m$ is homogeneous [12] if it preserves scalar multiplication: for all $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p$ and $c \in \mathbb{R}$,

\[f(c\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = c f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})\]

Example 15 (Homogeneous functions, and one that is not)

Example 15 (Homogeneous functions, and one that is not)

$f(x) = 2x$ on $\mathbb{R}$ is homogeneous. For $x, c \in \mathbb{R}$:

\[ \begin{aligned} f(c x) &= 2 c x \\ &= c (2x) \\ &= c f(x) \end{aligned} \]
$h(x) = x^2$ on $\mathbb{R}$ is not homogeneous, since $h(2 \cdot 1) = 4$ but $2\,h(1) = 2$.
$k(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathopen{}\left(x_1^3 + x_2^3\right)\mathclose{}^{1/3}$ on $\mathbb{R}^2$ is homogeneous but not additive. For $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^2$ and $c \in \mathbb{R}$:

\[ \begin{aligned} k(c \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= \mathopen{}\left(c^3 x_1^3 + c^3 x_2^3\right)\mathclose{}^{1/3} \\ &= \mathopen{}\left(c^3 \mathopen{}\left(x_1^3 + x_2^3\right)\mathclose{}\right)\mathclose{}^{1/3} \\ &= c \mathopen{}\left(x_1^3 + x_2^3\right)\mathclose{}^{1/3} \\ &= c\, k(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) \end{aligned} \]

but $k((1, 0) + (0, 1)) = k((1, 1)) = 2^{1/3}$, while $k((1, 0)) + k((0, 1)) = 2$.

Exercise 30 (Testing whether the absolute value is homogeneous)

Exercise 30 (Testing whether the absolute value is homogeneous) A homogeneous function turns a rescaled input into the same rescaling of its output, including a rescaling by a negative number. Let $f: \mathbb{R}\to \mathbb{R}$ be the absolute value, $f(x) = \mathopen{}\left|x\right|\mathclose{}$.

Is $f$ homogeneous?

Hint

Try a negative $c$ in Definition 19.

Solution 30

Solution 30. No. Take $c = -1$ and $x = 1$. Then

\[ f(c x) = \mathopen{}\left|-1\right|\mathclose{} = 1, \qquad c f(x) = -1 \cdot\mathopen{}\left|1\right|\mathclose{} = -1. \]

These differ, so $f$ is not homogeneous. It does satisfy $f(c x) = c f(x)$ for every $c \ge 0$, since then $\mathopen{}\left|c x\right|\mathclose{} = c \mathopen{}\left|x\right|\mathclose{}$.

Definition 20 (Linear map, linear transformation, linear function)

Definition 20 (Linear map, linear transformation, linear function) A function $f: \mathbb{R}^p \to \mathbb{R}^m$ is a linear map [12] (also called a linear transformation or linear function) if it is both additive and homogeneous.

Example 16 (A linear map, and two functions that are not)

Example 16 (A linear map, and two functions that are not)

$f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (2 x_1,\ x_1 + x_2)$ on $\mathbb{R}^2$ is linear. For $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^2$ and $c \in \mathbb{R}$:

\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) &= (2 (x_1 + y_1),\ (x_1 + y_1) + (x_2 + y_2)) \\ &= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) + f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) \end{aligned} \]

and

\[ \begin{aligned} f(c \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= (2 c x_1,\ c x_1 + c x_2) \\ &= c f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}). \end{aligned} \]
$g(x) = x + 1$ on $\mathbb{R}$ is not linear: additivity fails, since $g(0 + 0) = 1$ but $g(0) + g(0) = 2$.
$h(x) = x^2$ on $\mathbb{R}$ is not linear: $h(2 \cdot 1) = 4$ but $2\,h(1) = 2$, so $h(c x) \ne c\,h(x)$ for $c = 2$, $x = 1$.

Exercise 31 (Testing whether a difference of scaled measurements is a linear map)

Exercise 31 (Testing whether a difference of scaled measurements is a linear map) A score that scales two measurements and takes their difference, such as $3 x_1 - x_2$, is a common building block of a model. Let $f: \mathbb{R}^2 \to \mathbb{R}$ be $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 3 x_1 - x_2$.

Is $f$ a linear map?

Hint

Check the two conditions in Definition 20 one at a time, for any $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^2$ and $c \in \mathbb{R}$.

Solution 31

Solution 31. Yes. For any $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^2$, $f$ is additive:

\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) &= 3 (x_1 + y_1) - (x_2 + y_2) && \text{(}\href{linear-algebra-vectors.qmd#def-vector-addition}{\text{Definition~7 in Vectors}}\text{)} \\&= (3 x_1 - x_2) + (3 y_1 - y_2) && \text{(rearranging the terms)} \\&= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) + f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}). \end{aligned} \]

For any $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^2$ and $c \in \mathbb{R}$, $f$ is homogeneous:

\[ \begin{aligned} f(c \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= 3 c x_1 - c x_2 && \text{(}\href{#def-scalar-mult}{\text{Definition~9}}\text{)} \\&= c (3 x_1 - x_2) && \text{(factoring out } c \text{)} \\&= c f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}). \end{aligned} \]

Since $f$ is both, it is a linear map.

Definition 21 (Linear operator)

Definition 21 (Linear operator) A linear operator on $\mathbb{R}^p$ is a linear map (Definition 20) from $\mathbb{R}^p$ to itself, that is, a linear map $f : \mathbb{R}^p \to \mathbb{R}^p$ whose input and output have the same length. Some authors use “operator” more loosely, for any linear map.

Example 17 (A linear operator, and a linear map that is not one)

Example 17 (A linear operator, and a linear map that is not one) The map $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (2 x_1,\ x_1 + x_2)$ from Example 16 sends $\mathbb{R}^2$ to $\mathbb{R}^2$, so it is a linear operator on $\mathbb{R}^2$. The linear map $g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = x_1 + x_2$ sends $\mathbb{R}^2$ to $\mathbb{R}$, so it is linear but not an operator: its outputs have length $1$, not $2$.

Exercise 32 (Testing whether a projection is a linear operator)

Exercise 32 (Testing whether a projection is a linear operator) Dropping the last coordinate of a point in $\mathbb{R}^3$ projects it onto a plane, as a shadow does. Let $f: \mathbb{R}^3 \to \mathbb{R}^2$ be $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (x_1, x_2)$. This $f$ is a linear map.

Is $f$ a linear operator on $\mathbb{R}^3$?

Hint

Write $f$ as $f: \mathbb{R}^p \to \mathbb{R}^m$, and compare $p$ with $m$ as in Definition 21.

Solution 32

Solution 32. No. A linear operator on $\mathbb{R}^3$ sends $\mathbb{R}^3$ to $\mathbb{R}^3$. The outputs of $f$ lie in $\mathbb{R}^2$, not $\mathbb{R}^3$, so $f$ is a linear map but not a linear operator.

Theorem 11 (A linear map sends zero to zero)

Theorem 11 (A linear map sends zero to zero) If $f: \mathbb{R}^p \to \mathbb{R}^m$ is a linear map (Definition 20), then $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$.

Proof

Proof. On the left, $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$ is the zero vector of length $p$; on the right, it is the zero vector of length $m$.

\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) &= f(0 \cdot\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) && \text{(every entry of } 0 \cdot\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\text{ is } 0 \cdot 0 = 0 \text{; }\href{#def-scalar-mult}{\text{Definition~9}}\text{)} \\&= 0 \cdot f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) && \text{(homogeneity, with } c = 0 \text{ and } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\text{)} \\&= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}} && \text{(every entry of } 0 \cdot f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) \text{ is } 0 \text{; }\href{#def-scalar-mult}{\text{Definition~9}}\text{)} \end{aligned} \]

Example 18 (Ruling out linearity at zero)

Example 18 (Ruling out linearity at zero) In Example 16, $g(x) = x + 1$ has $g(0) = 1 \neq 0$, so by Theorem 11 it is not linear, without checking additivity or homogeneity.

Sending $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$ to $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$ is necessary but not sufficient: $h(x) = x^2$ in the same example has $h(0) = 0$ and is still not linear.

Exercise 33 (Ruling out a linear map from its value at zero)

Exercise 33 (Ruling out a linear map from its value at zero) Theorem 11 gives a quick test: a function that does not send $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$ to $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$ cannot be a linear map. Let $g: \mathbb{R}^2 \to \mathbb{R}^2$ be

\[ g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (x_1 + x_2,\ x_2 - 1). \]

Is $g$ a linear map?

Hint

Compute $g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}})$.

Solution 33

Solution 33. No.

\[ g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = (0 + 0,\ 0 - 1) = (0, -1) \ne \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}. \]

Every linear map sends $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$ to $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$ by Theorem 11, so $g$ is not one.

Theorem 12 (Multiplying by a matrix is a linear map)

Theorem 12 (Multiplying by a matrix is a linear map) For any $m \times p$ matrix $\mathbf{A}$, the function $f: \mathbb{R}^p \to \mathbb{R}^m$ defined by $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ is a linear map (Definition 20).

Proof

Proof. Both conditions are checked one entry at a time. For each $i \in \{1, \ldots, m\}$, additivity:

\[ \begin{aligned} \mathopen{}\left(\mathbf{A}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})\right)\mathclose{}_i &= \sum_{j=1}^pa_{ij} (\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})_j && \text{(}\href{#def-matvec-mult}{\text{Definition~16}}\text{)} \\&= \sum_{j=1}^pa_{ij} (x_j + y_j) && \text{(}\href{linear-algebra-vectors.qmd#def-vector-addition}{\text{Definition~7 in Vectors}}\text{)} \\&= \sum_{j=1}^p(a_{ij} x_j + a_{ij} y_j) && \text{(multiplication distributes over addition)} \\&= \sum_{j=1}^pa_{ij} x_j + \sum_{j=1}^pa_{ij} y_j && \text{(a sum of sums splits)} \\&= (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i + (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})_i && \text{(}\href{#def-matvec-mult}{\text{Definition~16}}\text{)} \\&= (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})_i && \text{(}\href{linear-algebra-vectors.qmd#def-vector-addition}{\text{Definition~7 in Vectors}}\text{)} \end{aligned} \]

and homogeneity:

\[ \begin{aligned} \mathopen{}\left(\mathbf{A}(c\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})\right)\mathclose{}_i &= \sum_{j=1}^pa_{ij} (c\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_j && \text{(}\href{#def-matvec-mult}{\text{Definition~16}}\text{)} \\&= \sum_{j=1}^pa_{ij}\, c\, x_j && \text{(}\href{#def-scalar-mult}{\text{Definition~9}}\text{)} \\&= \sum_{j=1}^pc\, a_{ij} x_j && \text{(products are symmetric)} \\&= c \sum_{j=1}^pa_{ij} x_j && \text{(a constant factor comes out of a sum)} \\&= c\, (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i && \text{(}\href{#def-matvec-mult}{\text{Definition~16}}\text{)} \\&= (c\, \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i && \text{(}\href{#def-scalar-mult}{\text{Definition~9}}\text{)} \end{aligned} \]

The four algebra rules cited here are on the algebra page: distributivity, a sum of sums, symmetry of products, and a constant factor.

Exercise 34 (Recognizing a linear map as multiplication by a matrix)

Exercise 34 (Recognizing a linear map as multiplication by a matrix) Theorem 12 saves checking additivity and homogeneity by hand: a function that multiplies its input by a fixed matrix is a linear map. Let $f: \mathbb{R}^2 \to \mathbb{R}^2$ be

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (x_1 + 2 x_2,\ 3 x_1 - x_2). \]

Is $f$ a linear map?

Hint

Find a $2 \times 2$ matrix $\mathbf{A}$ with $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$, using Definition 16.

Solution 34

Solution 34. Yes. Let

\[ \mathbf{A} = \begin{bmatrix} 1 & 2 \\ 3 & -1 \end{bmatrix}. \]

By Definition 16,

\[ \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \begin{bmatrix} 1 \cdot x_1 + 2 \cdot x_2 \\ 3 \cdot x_1 + (-1) \cdot x_2 \end{bmatrix} = \begin{bmatrix} x_1 + 2 x_2 \\ 3 x_1 - x_2 \end{bmatrix} = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}). \]

So $f$ multiplies its input by $\mathbf{A}$, and $f$ is a linear map by Theorem 12.

Theorem 13 (A linear map preserves linear combinations)

Theorem 13 (A linear map preserves linear combinations) If $f: \mathbb{R}^p \to \mathbb{R}^m$ is a linear map (Definition 20), then for any vectors $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1, \ldots, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_k \in \mathbb{R}^p$ and any coefficients $c_1, \ldots, c_k \in \mathbb{R}$:

\[f\mathopen{}\left(\sum_{j=1}^{k} c_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j\right)\mathclose{} = \sum_{j=1}^{k} c_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j)\]

Proof

Proof. By induction on $k$. For $k = 1$, the claim $f(c_1 \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1) = c_1 f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1)$ is homogeneity.

Suppose the claim holds for $k$ vectors. Then for $k + 1$:

\[ \begin{aligned} f\mathopen{}\left(\sum_{j=1}^{k+1} c_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j\right)\mathclose{} &= f\mathopen{}\left(\sum_{j=1}^{k} c_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j + c_{k+1} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_{k+1}\right)\mathclose{} && \text{(split off the last term)} \\&= f\mathopen{}\left(\sum_{j=1}^{k} c_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j\right)\mathclose{} + f(c_{k+1} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_{k+1}) && \text{(additivity)} \\&= \sum_{j=1}^{k} c_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j) + f(c_{k+1} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_{k+1}) && \text{(induction hypothesis)} \\&= \sum_{j=1}^{k} c_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j) + c_{k+1} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_{k+1}) && \text{(homogeneity)} \\&= \sum_{j=1}^{k+1} c_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j) && \text{(join the last term back on)} \end{aligned} \]

Exercise 35 (Evaluating a linear map on a linear combination)

Exercise 35 (Evaluating a linear map on a linear combination) A linear map is determined on a linear combination of vectors by its values on those vectors alone. Let $f: \mathbb{R}^2 \to \mathbb{R}^2$ be a linear map with

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1) = \begin{bmatrix} 1 \\ 2 \end{bmatrix}, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_2) = \begin{bmatrix} 0 \\ -1 \end{bmatrix}. \]

Find $f(3 \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1 - 2 \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_2)$.

Hint

Apply Theorem 13 with $c_1 = 3$ and $c_2 = -2$.

Solution 35

Solution 35. \[ \begin{aligned} f(3 \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1 - 2 \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_2) &= 3 f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1) - 2 f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_2) && \text{(}\href{#thm-linear-map-lincom}{\text{Theorem~13}}\text{)} \\&= 3 \begin{bmatrix} 1 \\ 2 \end{bmatrix} - 2 \begin{bmatrix} 0 \\ -1 \end{bmatrix} && \text{(the given values)} \\&= \begin{bmatrix} 3 \\ 6 \end{bmatrix} - \begin{bmatrix} 0 \\ -2 \end{bmatrix} && \text{(}\href{#def-scalar-mult}{\text{Definition~9}}\text{)} \\&= \begin{bmatrix} 3 \\ 8 \end{bmatrix} && \text{(subtracting entry by entry)} \end{aligned} \]

Theorem 14 (Every linear map is multiplication by a matrix)

Theorem 14 (Every linear map is multiplication by a matrix) If $f: \mathbb{R}^p \to \mathbb{R}^m$ is a linear map (Definition 20), then there is exactly one $m \times p$ matrix $\mathbf{A}$ such that

\[f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\quad \text{for all } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\]

Column $j$ of $\mathbf{A}$ is $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)$, the value of $f$ at the $j$th indicator vector (Definition 19 in Vectors); that is, $a_{ij} = \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i$.

Proof

Proof. Existence. Let $\mathbf{A}$ be the $m \times p$ matrix with $a_{ij} = \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i$. For any $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p$,

\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= f\mathopen{}\left(\sum_{j=1}^px_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\right)\mathclose{} && \text{(}\href{linear-algebra-vectors.qmd#thm-standard-basis-expansion}{\text{Theorem~4 in Vectors}}\text{)} \\&= \sum_{j=1}^px_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j) && \text{(}\href{#thm-linear-map-lincom}{\text{Theorem~13}}\text{)} \end{aligned} \]

and entry $i$ of that sum is

\[ \begin{aligned} \mathopen{}\left(\sum_{j=1}^px_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i &= \sum_{j=1}^px_j \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i && \text{(}\href{linear-algebra-vectors.qmd#def-linear-combination}{\text{Definition~9 in Vectors}}\text{, entry by entry)} \\&= \sum_{j=1}^px_j a_{ij} && \text{(definition of } \mathbf{A} \text{)} \\&= \sum_{j=1}^pa_{ij} x_j && \text{(products are symmetric)} \\&= (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i && \text{(}\href{#def-matvec-mult}{\text{Definition~16}}\text{)} \end{aligned} \]

Uniqueness. Suppose $\mathbf{B}$ is any $m \times p$ matrix with $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ for all $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$. Taking $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j$, for each $i$:

\[ \begin{aligned} \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i &= (\mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)_i && \text{(assumption on } \mathbf{B} \text{)} \\&= \sum_{k=1}^{p} b_{ik} (\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)_k && \text{(}\href{#def-matvec-mult}{\text{Definition~16}}\text{)} \\&= b_{ij} \cdot 1 + \sum_{k \neq j} b_{ik} \cdot 0 && \text{(}\href{linear-algebra-vectors.qmd#def-indicator-vector}{\text{Definition~19 in Vectors}}\text{, splitting off the } k = j \text{ term)} \\&= b_{ij} + 0 && \text{(times } 1 \text{ changes nothing; times } 0 \text{ gives } 0 \text{)} \\&= b_{ij} && \text{(adding } 0 \text{ changes nothing)} \end{aligned} \]

\[ \begin{aligned} b_{ij} &= \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i \\ &= a_{ij} \end{aligned} \]

for every $i$ and $j$, and $\mathbf{B} = \mathbf{A}$.

Definition 22 (Matrix of a linear map)

Definition 22 (Matrix of a linear map) The matrix of a linear map $f: \mathbb{R}^p \to \mathbb{R}^m$ is the unique $m \times p$ matrix $\mathbf{A}$ with $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ for all $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ (Theorem 14). Its column $j$ is $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)$.

Example 19 (Finding the matrix of a linear map)

Example 19 (Finding the matrix of a linear map) For $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (2 x_1,\ x_1 + x_2)$ from Example 16, the values of $f$ at the indicator vectors are

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_1) = f\mathopen{}\left(\begin{bmatrix}1 \\ 0\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}2 \cdot 1 \\ 1 + 0\end{bmatrix} = \begin{bmatrix}2 \\ 1\end{bmatrix}, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_2) = f\mathopen{}\left(\begin{bmatrix}0 \\ 1\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}2 \cdot 0 \\ 0 + 1\end{bmatrix} = \begin{bmatrix}0 \\ 1\end{bmatrix} \]

and these are the columns of the matrix of $f$ (Definition 22): $\mathbf{A} = \begin{bmatrix}2 & 0 \\ 1 & 1\end{bmatrix}$.

Exercise 36 (Evaluating a linear map from its values on the indicator vectors)

Exercise 36 (Evaluating a linear map from its values on the indicator vectors) By Theorem 14, the values of a linear map at the indicator vectors determine it everywhere. Let $f: \mathbb{R}^2 \to \mathbb{R}^2$ be a linear map with

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_1) = \begin{bmatrix} 1 \\ 3 \end{bmatrix}, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_2) = \begin{bmatrix} -2 \\ 0 \end{bmatrix}. \]

Find $f\mathopen{}\left(\begin{bmatrix} 4 \\ 1 \end{bmatrix}\right)\mathclose{}$.

Hint

Build the matrix of $f$ from the two given values, then multiply.

Solution 36

Solution 36. By Theorem 14, $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$, where column $j$ of $\mathbf{A}$ is $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)$:

\[ \mathbf{A} = \begin{bmatrix} 1 & -2 \\ 3 & 0 \end{bmatrix}. \]

\[ f\mathopen{}\left(\begin{bmatrix} 4 \\ 1 \end{bmatrix}\right)\mathclose{} = \begin{bmatrix} 1 \cdot 4 + (-2) \cdot 1 \\ 3 \cdot 4 + 0 \cdot 1 \end{bmatrix} = \begin{bmatrix} 2 \\ 12 \end{bmatrix}. \]

Exercise 37 (Finding the matrix of a linear map into $\mathbb{R}^3$)

Exercise 37 (Finding the matrix of a linear map into $\mathbb{R}^3$) The matrix of a linear map turns the map into a table of numbers that software can store and multiply. Let $f: \mathbb{R}^2 \to \mathbb{R}^3$ be the linear map

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (x_2,\ x_1,\ x_1 - x_2). \]

Find the matrix of $f$.

Hint

Column $j$ of the matrix is $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)$.

Solution 37

Solution 37. The values of $f$ at the indicator vectors are

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_1) = f\mathopen{}\left(\begin{bmatrix} 1 \\ 0 \end{bmatrix}\right)\mathclose{} = \begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix}, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_2) = f\mathopen{}\left(\begin{bmatrix} 0 \\ 1 \end{bmatrix}\right)\mathclose{} = \begin{bmatrix} 1 \\ 0 \\ -1 \end{bmatrix}. \]

These values are the columns of the matrix of $f$:

\[ \mathbf{A} = \begin{bmatrix} 0 & 1 \\ 1 & 0 \\ 1 & -1 \end{bmatrix}. \]

It is $3 \times 2$, since $f$ sends $\mathbb{R}^2$ to $\mathbb{R}^3$.

Corollary 1 (Linear functions from vectors to numbers)

Corollary 1 (Linear functions from vectors to numbers) A function $f: \mathbb{R}^p \to \mathbb{R}$ is linear (Definition 20) if and only if there is a vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \in \mathbb{R}^p$ such that

\[f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} \quad \text{for all } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\]

and then $w_j = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)$ for each $j$.

Proof

Proof. If $f$ is linear. Theorem 14 with $m = 1$ gives a $1 \times p$ matrix $\mathbf{A}$ with $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ and $a_{1j} = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)$. Let $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}$ be the vector with $w_j = a_{1j}$, so that $\mathbf{A} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top}$. Then

\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(}\href{#thm-linear-map-matrix}{\text{Theorem~14}}\text{)} \\&= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && (\mathbf{A} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top}) \end{aligned} \]

and

\[ \begin{aligned} w_j &= a_{1j} \\ &= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j). \end{aligned} \]

If $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$. Here $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top}$ is a $1 \times p$ matrix, so $f$ is linear by Theorem 12.

Example 20 (A linear function from $\mathbb{R}^2$ to $\mathbb{R}$)

Example 20 (A linear function from $\mathbb{R}^2$ to $\mathbb{R}$) $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 3 x_1 + 5 x_2$ is $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ with $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} = {(3, 5)}^{\top}$, so it is linear by Corollary 1. The entries of $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}$ are the values at the indicator vectors:

\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_1) &= 3 \cdot 1 + 5 \cdot 0 \\ &= 3 \end{aligned} \]

and

\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_2) &= 3 \cdot 0 + 5 \cdot 1 \\ &= 5. \end{aligned} \]

$g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 3 x_1 + 5 x_2 + 1$ is not linear, because $g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = 1 \neq 0$ (Theorem 11).

Exercise 38 (Finding the weight vector of a linear score)

Exercise 38 (Finding the weight vector of a linear score) A linear score of $p$ measurements is a dot product of the measurements with a vector of weights, by Corollary 1. Let $f: \mathbb{R}^3 \to \mathbb{R}$ be a linear map with

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_1) = 2, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_2) = 0, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_3) = -1. \]

Find the vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}$ with $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ for every $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^3$.

Hint

Corollary 1 gives each entry $w_j$ of $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}$.

Solution 38

Solution 38. By Corollary 1, $w_j = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)$ for each $j$, so

\[ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} = \begin{bmatrix} 2 \\ 0 \\ -1 \end{bmatrix}. \]

Then $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 2 x_1 - x_3$ for every $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^3$.

Videos

Two chapters of the Essence of linear algebra series by 3Blue1Brown show the results above geometrically:

Linear transformations and matrices (Chapter 3) shows why a linear map is fixed by where it sends the indicator vectors, which become the columns of its matrix (Definition 22).
Dot products and duality (Chapter 9) shows why every linear map from $\mathbb{R}^p$ to $\mathbb{R}$ is a dot product with a fixed vector (Corollary 1).

Definition 23 (Translation)

Definition 23 (Translation) For a fixed vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \in \mathbb{R}^m$, the translation [13] by $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ is the function $t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}} : \mathbb{R}^m \to \mathbb{R}^m$ defined by

\[t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\]

Example 21 (Translating two points)

Example 21 (Translating two points) The translation by $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = {(1, -2)}^{\top}$ sends

\[ t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}\mathopen{}\left(\begin{bmatrix}0 \\ 0\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}0 + 1 \\ 0 + (-2)\end{bmatrix} = \begin{bmatrix}1 \\ -2\end{bmatrix}, \qquad t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}\mathopen{}\left(\begin{bmatrix}3 \\ 4\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}3 + 1 \\ 4 + (-2)\end{bmatrix} = \begin{bmatrix}4 \\ 2\end{bmatrix} \]

Both points move one unit right and two units down.

Remark 5 (A translation moves every point by the same vector)

Remark 5 (A translation moves every point by the same vector). A translation by $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ moves every point by the same vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$.

Exercise 39 (Finding the translation that moves one point to another)

Exercise 39 (Finding the translation that moves one point to another) A translation moves every point by the same vector, so where it sends one point fixes the whole translation. A translation $t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}$ of $\mathbb{R}^2$ sends ${(2, 3)}^{\top}$ to ${(5, 1)}^{\top}$.

Find $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$.

Hint

Solve ${(2, 3)}^{\top} + \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = {(5, 1)}^{\top}$ entry by entry.

Solution 39

Solution 39. By Definition 23, $t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}\mathopen{}\left({(2, 3)}^{\top}\right)\mathclose{} = {(2, 3)}^{\top} + \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$. Matching this vector with ${(5, 1)}^{\top}$ entry by entry gives

\[ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \begin{bmatrix}5 - 2 \\ 1 - 3\end{bmatrix} = \begin{bmatrix}3 \\ -2\end{bmatrix}. \]

Definition 24

Definition 24

Affine map, affine transformation, affine function

A function $f: \mathbb{R}^p \to \mathbb{R}^m$ is an affine map (also called an affine transformation or affine function) if it is a matrix product followed by a translation (Definition 23) (a composition, with the matrix product as the inner function): there are an $m \times p$ matrix $\mathbf{A}$ and a vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \in \mathbb{R}^m$ such that

\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}(\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) \\ &= \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \quad \text{for all } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p \end{aligned} \]

Exercise 40 (Testing whether a rescale-and-shift is an affine map)

Exercise 40 (Testing whether a rescale-and-shift is an affine map) Rescaling a measurement and then shifting it, as when converting units with a different zero point, is the typical affine map. Let $f: \mathbb{R}^2 \to \mathbb{R}^2$ be

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (2 x_1 + 1,\ x_2 - 3). \]

Is $f$ an affine map?

Hint

Look for a $2 \times 2$ matrix $\mathbf{A}$ and a vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ with $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$.

Solution 40

Solution 40. Yes. Let

\[ \mathbf{A} = \begin{bmatrix}2 & 0 \\ 0 & 1\end{bmatrix}, \qquad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \begin{bmatrix}1 \\ -3\end{bmatrix}. \]

By Definition 16, $\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= {(2 x_1, x_2)}^{\top}$, so

\[ \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \begin{bmatrix}2 x_1 + 1 \\ x_2 - 3\end{bmatrix} = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) \]

for every $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^2$. So $f$ is an affine map.

Definition 25 (Offset of an affine map, intercept, bias)

Definition 25 (Offset of an affine map, intercept, bias) In Definition 24, the vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ is called the offset of $f$ (statistics also calls it the intercept, and machine learning the bias).

Remark 6 (Real-valued affine maps)

Remark 6 (Real-valued affine maps). For a real-valued function (Definition 4 in Vectors, $m = 1$), $\mathbf{A}$ is a single row, written $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top}$, and $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ is a single number $b$, so an affine function is a linear function $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ (Corollary 1) plus an offset:

\[f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b, \qquad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \in \mathbb{R}^p, \quad b \in \mathbb{R} \tag{1}\]

Exercise 41 (Reading off the offset of an affine map)

Exercise 41 (Reading off the offset of an affine map) In a fitted model, the offset is the part of the output that does not depend on the input. Let $f: \mathbb{R}^2 \to \mathbb{R}^2$ be the affine map

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (x_1 - x_2 + 4,\ 3 x_2 - 1). \]

Find the offset of $f$.

Hint

Write $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})$ as $\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ as in Definition 24.

Solution 41

Solution 41. \[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \begin{bmatrix}1 & -1 \\ 0 & 3\end{bmatrix} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \begin{bmatrix}4 \\ -1\end{bmatrix}, \]

so the offset of $f$ is $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = {(4, -1)}^{\top}$. Each entry of $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ is the term with no $x_j$ in the matching entry of $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})$.

Definition 26 (Augmented vector)

Definition 26 (Augmented vector) The augmented vector of $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p$ is the vector ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} \in \mathbb{R}^{p+1}$ formed by placing a $1$ above the entries of $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$:

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} &= \begin{bmatrix}1 \\ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\end{bmatrix} \\ &= {(1, x_1, \ldots, x_p)}^{\top} \end{aligned} \]

Example 22 (An affine function as a dot product with an augmented vector)

Example 22 (An affine function as a dot product with an augmented vector) For $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= {(2, 5)}^{\top}$, the augmented vector is ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} = {(1, 2, 5)}^{\top}$. With $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} = {(3, -1)}^{\top}$ and $b = 4$, the affine function $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b$ (Equation 1) equals the dot product of ${(b, w_1, w_2)}^{\top} = {(4, 3, -1)}^{\top}$ with ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}}$:

\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b &= 3 \cdot 2 + (-1) \cdot 5 + 4 \\ &= 5, \\ {(4, 3, -1)}^{\top} \cdot {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} &= 4 \cdot 1 + 3 \cdot 2 + (-1) \cdot 5 \\ &= 5 \end{aligned} \]

Exercise 42 (Absorbing an offset into a dot product with the augmented vector)

Exercise 42 (Absorbing an offset into a dot product with the augmented vector) The augmented vector lets software treat an affine score as a dot product, with the offset as one more weight. Let $f: \mathbb{R}^2 \to \mathbb{R}$ be

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 2 x_1 - x_2 + 5. \]

Find the vector $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \in \mathbb{R}^3$ with $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \cdot {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}}$ for every $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^2$.

Hint

Follow Example 22. The first entry of ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}}$ is $1$.

Solution 42

Solution 42. Let $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = {(5, 2, -1)}^{\top}$. Since ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} = {(1, x_1, x_2)}^{\top}$ by Definition 26,

\[ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \cdot {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} = 5 \cdot 1 + 2 x_1 + (-1) x_2 = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}). \]

The first entry of $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}$ is the offset $5$, and the other two are the numbers multiplying $x_1$ and $x_2$.

Theorem 15 (An affine map sends zero to its offset)

Theorem 15 (An affine map sends zero to its offset) An affine map $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ (Definition 24) satisfies $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$.

Proof

Proof. First, $\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$: for each $i$,

\[ \begin{aligned} (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}})_i &= \sum_{j=1}^pa_{ij} \cdot 0 && \text{(}\href{#def-matvec-mult}{\text{Definition~16}}\text{)} \\&= \sum_{j=1}^p0 && \text{(any number times } 0 \text{ is } 0 \text{)} \\&= 0 && \text{(a sum of zeros is } 0 \text{)} \end{aligned} \]

Then

\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} && \text{(}\href{#def-affine-map}{\text{Definition~24}}\text{)} \\&= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} && (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) \\&= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} && \text{(adding } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\text{ changes no entry; }\href{linear-algebra-vectors.qmd#def-vector-addition}{\text{Definition~7 in Vectors}}\text{)} \end{aligned} \]

Exercise 43 (Recovering a column of an affine map’s matrix from two values)

Exercise 43 (Recovering a column of an affine map’s matrix from two values) Theorem 15 says that the value of an affine map at $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$ is its offset, so subtracting that value isolates the matrix part. Let $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ be an affine map from $\mathbb{R}^2$ to $\mathbb{R}^2$ with

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \begin{bmatrix}1 \\ 2\end{bmatrix}, \qquad f\mathopen{}\left(\begin{bmatrix}1 \\ 0\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}4 \\ 2\end{bmatrix}. \]

Find the first column of $\mathbf{A}$.

Hint

$\mathbf{A} \begin{bmatrix}1 \\ 0\end{bmatrix}$ is the first column of $\mathbf{A}$.

Solution 43

Solution 43. By Theorem 15, $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = {(1, 2)}^{\top}$. So

\[ \begin{aligned} \mathbf{A} \begin{bmatrix}1 \\ 0\end{bmatrix} &= f\mathopen{}\left(\begin{bmatrix}1 \\ 0\end{bmatrix}\right)\mathclose{} - \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} && \text{(}\href{#def-affine-map}{\text{Definition~24}}\text{)} \\&= \begin{bmatrix}4 - 1 \\ 2 - 2\end{bmatrix} && \text{(the given values)} \\&= \begin{bmatrix}3 \\ 0\end{bmatrix}. \end{aligned} \]

By Definition 16, $\mathbf{A} {(1, 0)}^{\top}$ is the first column of $\mathbf{A}$, which is therefore ${(3, 0)}^{\top}$.

Theorem 16 (An affine map is linear exactly when its offset is zero)

Theorem 16 (An affine map is linear exactly when its offset is zero) An affine map $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ (Definition 24) is linear (Definition 20) if and only if $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$.

Proof

Proof. If $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$, then

\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\\ &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \end{aligned} \]

which is linear by Theorem 12. Conversely, if $f$ is linear, then $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$ by Theorem 11, and $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ by Theorem 15, so $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$.

Example 23 (Two affine maps with the same matrix)

Example 23 (Two affine maps with the same matrix) Let $\mathbf{A} = \begin{bmatrix}2 & 0 \\ 1 & 1\end{bmatrix}$, the matrix of the linear map in Example 16. $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ has offset $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$, so it is linear. $g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \begin{bmatrix}1 \\ 0\end{bmatrix}$ has offset $\begin{bmatrix}1 \\ 0\end{bmatrix} \neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$, so by Theorem 16 it is not linear: indeed $g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \begin{bmatrix}1 \\ 0\end{bmatrix}$, not $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$.

Exercise 44 (Choosing a constant that makes an affine map linear)

Exercise 44 (Choosing a constant that makes an affine map linear) By Theorem 16, an affine map is a linear map exactly when its offset is $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$. For $c \in \mathbb{R}$, let $f: \mathbb{R}^2 \to \mathbb{R}^2$ be

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (x_1 - x_2 + c - 2,\ 3 x_1 + c^2 - 4). \]

For which values of $c$ is $f$ linear?

Hint

Find the offset of $f$ as a function of $c$.

Solution 44

Solution 44. Only for $c = 2$. The map is

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \begin{bmatrix}1 & -1 \\ 3 & 0\end{bmatrix} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \begin{bmatrix}c - 2 \\ c^2 - 4\end{bmatrix}, \]

so its offset is ${(c - 2,\ c^2 - 4)}^{\top}$. By Theorem 16, $f$ is linear exactly when both entries are $0$. The first entry is $0$ only for $c = 2$, and then the second entry is $2^2 - 4 = 0$ too.

Corollary 2 (Every linear map is affine)

Corollary 2 (Every linear map is affine) Every linear map $f : \mathbb{R}^p \to \mathbb{R}^m$ (Definition 20) is an affine map (Definition 24) with offset $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$.

Proof

Proof. If $f$ is linear, Theorem 14 gives a matrix $\mathbf{A}$ with

\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\\ &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}, \end{aligned} \]

which is Definition 24 with $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$.

Exercise 45 (Testing whether a linear score is an affine map)

Exercise 45 (Testing whether a linear score is an affine map) Some results hold only for affine maps, so it helps to check whether a given score qualifies. Let $f: \mathbb{R}^2 \to \mathbb{R}$ be $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = x_1 - 4 x_2$, a linear map.

Is $f$ an affine map?

Hint

Use Corollary 2.

Solution 45

Solution 45. Yes. By Corollary 2, every linear map is an affine map with offset $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$. Here

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \begin{bmatrix}1 & -4\end{bmatrix} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ 0, \]

with the $1 \times 2$ matrix $\begin{bmatrix}1 & -4\end{bmatrix}$ and offset $0$.

Theorem 17 (An affine map determines its matrix and offset)

Theorem 17 (An affine map determines its matrix and offset) If $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ and $f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}'\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}'$ for all $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p$ (Definition 24), then $\mathbf{A}' = \mathbf{A}$ and $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}' = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$.

Proof

Proof. By Theorem 15, applied to each formula,

\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} &= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) \\ &= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}'. \end{aligned} \]

Then

\[ \begin{aligned} \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}&= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) - \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \\ &= \mathbf{A}'\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} \end{aligned} \]

for all $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$, so $\mathbf{A}$ and $\mathbf{A}'$ are both matrices (Definition 22) of the linear map $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\mapsto f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) - \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}$ (linear by Theorem 12), and the uniqueness in Theorem 14 gives $\mathbf{A}' = \mathbf{A}$.

Exercise 46 (Matching the offsets of two formulas for one affine map)

Exercise 46 (Matching the offsets of two formulas for one affine map) Theorem 17 says that an affine map has only one matrix and one offset, so two ways of writing the same map must agree term by term. For $c \in \mathbb{R}$, suppose that

\[ \begin{bmatrix}1 & 0 \\ 2 & 1\end{bmatrix} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \begin{bmatrix}3 \\ 1\end{bmatrix} = \begin{bmatrix}1 & 0 \\ 2 & 1\end{bmatrix} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \begin{bmatrix}c \\ 1\end{bmatrix} \quad \text{for every } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^2. \]

Find $c$.

Hint

Compare the offsets of the two sides.

Solution 46

Solution 46. $c = 3$. Both sides describe the same affine map. By Theorem 17, the two sides have the same offset, so ${(3, 1)}^{\top} = {(c, 1)}^{\top}$ and $c = 3$. Setting $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}$ gives the same answer directly.

Remark 7 (“Linear” in this site, and in machine learning and statistics)

Remark 7 (“Linear” in this site, and in machine learning and statistics). This site uses “linear” only in the sense of Definition 20, so by Theorem 16 a function with a nonzero offset is affine but not linear. For functions of one variable, this is the same split as affine $a x + b$ versus linear $a x$ on the algebra page. (The algebra page writes these as $m x + b$ and $m x$; here $m$ already names the number of outputs.)

Machine learning and statistics often call any function of the form Equation 1 “linear”, as in a linear model in statistics or a linear layer in a neural network, and call the offset $b$ the bias (machine learning) or the intercept (statistics), as Definition 24 notes. A model with an intercept is affine in $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ but linear in its coefficients $(b, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}})$, and that is the sense of “linear” in “linear model”: it stays linear in the coefficients even when the inputs are transformed: $w_1 x + w_2 x^2 + b$ is not affine in $x$, but it is linear in $(b, w_1, w_2)$. Separately, the affine function $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b$ is a linear function of the augmented vector ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}}$ (Definition 26), with $b$ as the coefficient on its first entry.

symbol	meaning	units
\(x_1\)	age	years
\(x_2\)	systolic blood pressure	mmHg
\(x_3\)	body mass index	kg/m\(^2\)

8 Transposes of sums and products

Theorem 18 (Transpose of a sum)

Theorem 18 (Transpose of a sum) For $m \times n$ matrices $\mathbf{A}$ and $\mathbf{B}$:

\[{(\mathbf{A} + \mathbf{B})}^{\top} = {\mathbf{A}}^{\top} + {\mathbf{B}}^{\top}\]

In particular, for column vectors $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}$ and $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}$ of the same length:

\[{(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})}^{\top} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} + {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}}^{\top}\]

Proof

Proof. Entry $(i, j)$ of each side:

\[ \begin{aligned} \mathopen{}\left[{(\mathbf{A} + \mathbf{B})}^{\top}\right]\mathclose{}_{ij} &= (\mathbf{A} + \mathbf{B})_{ji} && \text{(definition of the transpose)} \\ &= a_{ji} + b_{ji} && \text{(definition of matrix addition)} \\ &= ({\mathbf{A}}^{\top})_{ij} + ({\mathbf{B}}^{\top})_{ij} && \text{(definition of the transpose)} \\ &= \mathopen{}\left[{\mathbf{A}}^{\top} + {\mathbf{B}}^{\top}\right]\mathclose{}_{ij} && \text{(definition of matrix addition)} \end{aligned} \]

Exercise 52 (Transposing a sum of two matrices)

Exercise 52 (Transposing a sum of two matrices) Theorem 18 lets a transpose be taken before or after adding, whichever is less work. Let

\[ \mathbf{A} = \begin{bmatrix}1 & 2 & 0 \\ 3 & -1 & 4\end{bmatrix}, \qquad \mathbf{B} = \begin{bmatrix}0 & 1 & 1 \\ 2 & 2 & -3\end{bmatrix}. \]

Find ${(\mathbf{A} + \mathbf{B})}^{\top}$.

Hint

Add first with Definition 8, then take the transpose.

Solution 52

Solution 52. Adding entry by entry,

\[ \mathbf{A} + \mathbf{B} = \begin{bmatrix}1 & 3 & 1 \\ 5 & 1 & 1\end{bmatrix}, \]

\[ {(\mathbf{A} + \mathbf{B})}^{\top} = \begin{bmatrix}1 & 5 \\ 3 & 1 \\ 1 & 1\end{bmatrix}. \]

By Theorem 18, adding ${\mathbf{A}}^{\top} = \begin{bmatrix}1 & 3 \\ 2 & -1 \\ 0 & 4\end{bmatrix}$ and ${\mathbf{B}}^{\top} = \begin{bmatrix}0 & 2 \\ 1 & 2 \\ 1 & -3\end{bmatrix}$ gives the same matrix.

Theorem 19 (Transpose of a product)

Theorem 19 (Transpose of a product) For an $m \times k$ matrix $\mathbf{A}$ and a $k \times n$ matrix $\mathbf{B}$:

\[ \underbrace{{(\mathbf{A}\mathbf{B})}^{\top}}_{n \times m} = \underbrace{{\mathbf{B}}^{\top}}_{n \times k}\,\underbrace{{\mathbf{A}}^{\top}}_{k \times m} \]

Proof

Proof. Entry $(i, j)$ of each side:

\[ \begin{aligned} \mathopen{}\left[{(\mathbf{A}\mathbf{B})}^{\top}\right]\mathclose{}_{ij} &= (\mathbf{A}\mathbf{B})_{ji} && \text{(definition of the transpose)} \\ &= \sum_{s=1}^{k} a_{js}\, b_{si} && \text{(definition of matrix multiplication)} \\ &= \sum_{s=1}^{k} ({\mathbf{B}}^{\top})_{is}\, ({\mathbf{A}}^{\top})_{sj} && \text{(definition of the transpose; reorder each product)} \\ &= \mathopen{}\left[{\mathbf{B}}^{\top}\,{\mathbf{A}}^{\top}\right]\mathclose{}_{ij} && \text{(definition of matrix multiplication)} \end{aligned} \]

Remark 8 (The order of the factors reverses)

Remark 8 (The order of the factors reverses). Transposing a product reverses the order of the factors. Keeping the original order usually gives an undefined product or a different matrix. For example, with $\mathbf{A} = [1,\; 2]$ ($1 \times 2$) and $\mathbf{B} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$ ($2 \times 1$),

\[ \begin{aligned} \mathbf{A}\mathbf{B} &= 1 \cdot 3 + 2 \cdot 4 \\ &= 11, \end{aligned} \]

so ${(\mathbf{A}\mathbf{B})}^{\top} = 11$, and

\[ {\mathbf{B}}^{\top}\,{\mathbf{A}}^{\top} = [3,\; 4] \begin{bmatrix} 1 \\ 2 \end{bmatrix} = 11, \qquad {\mathbf{A}}^{\top}\,{\mathbf{B}}^{\top} = \begin{bmatrix} 1 \\ 2 \end{bmatrix} [3,\; 4] = \begin{bmatrix} 3 & 4 \\ 6 & 8 \end{bmatrix}. \]

If instead $\mathbf{A}$ is $2 \times 3$ and $\mathbf{B}$ is $3 \times 1$, then ${\mathbf{B}}^{\top}\,{\mathbf{A}}^{\top}$ is a $(1 \times 3)(3 \times 2)$ product, but ${\mathbf{A}}^{\top}\,{\mathbf{B}}^{\top}$ is a $(3 \times 2)(1 \times 3)$ product, which is undefined.

Exercise 53 (The order of the factors in a transposed square product)

Exercise 53 (The order of the factors in a transposed square product) Remark 8 shows that, for factors that are not square, keeping the original order of the factors in the transpose of a product can give an undefined product. For two square matrices of the same size, both ${\mathbf{A}}^{\top}{\mathbf{B}}^{\top}$ and ${\mathbf{B}}^{\top}{\mathbf{A}}^{\top}$ are defined, so the shapes alone cannot settle which order gives ${(\mathbf{A}\mathbf{B})}^{\top}$. Let

\[ \mathbf{A} = \begin{bmatrix} 1 & 2 \\ 0 & 1 \end{bmatrix}, \qquad \mathbf{B} = \begin{bmatrix} 1 & 0 \\ 1 & 1 \end{bmatrix}. \]

Does ${\mathbf{A}}^{\top}{\mathbf{B}}^{\top}$ equal ${(\mathbf{A}\mathbf{B})}^{\top}$ for these $\mathbf{A}$ and $\mathbf{B}$?

Hint

Compute $\mathbf{A}\mathbf{B}$ with Definition 10 and take its transpose, then compute ${\mathbf{A}}^{\top}{\mathbf{B}}^{\top}$ and compare the two entry by entry.

Solution 53

Solution 53. No: the two matrices differ, so keeping the order of the factors gives the wrong transpose.

The product $\mathbf{A}\mathbf{B}$ and its transpose are

\[ \begin{aligned} \mathbf{A}\mathbf{B} &= \begin{bmatrix} 1 \cdot 1 + 2 \cdot 1 & 1 \cdot 0 + 2 \cdot 1 \\ 0 \cdot 1 + 1 \cdot 1 & 0 \cdot 0 + 1 \cdot 1 \end{bmatrix} && \text{(}\href{#def-matrix-mult}{\text{Definition~10}}\text{)} \\ &= \begin{bmatrix} 3 & 2 \\ 1 & 1 \end{bmatrix}, && \text{(multiply and add)} \end{aligned} \]

so ${(\mathbf{A}\mathbf{B})}^{\top} = \begin{bmatrix} 3 & 1 \\ 2 & 1 \end{bmatrix}$.

The transposes are ${\mathbf{A}}^{\top} = \begin{bmatrix} 1 & 0 \\ 2 & 1 \end{bmatrix}$ and ${\mathbf{B}}^{\top} = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}$, so

\[ \begin{aligned} {\mathbf{A}}^{\top}{\mathbf{B}}^{\top} &= \begin{bmatrix} 1 \cdot 1 + 0 \cdot 0 & 1 \cdot 1 + 0 \cdot 1 \\ 2 \cdot 1 + 1 \cdot 0 & 2 \cdot 1 + 1 \cdot 1 \end{bmatrix} && \text{(}\href{#def-matrix-mult}{\text{Definition~10}}\text{)} \\ &= \begin{bmatrix} 1 & 1 \\ 2 & 3 \end{bmatrix}. && \text{(multiply and add)} \end{aligned} \]

Entry $(1, 1)$ is $3$ in ${(\mathbf{A}\mathbf{B})}^{\top}$ and $1$ in ${\mathbf{A}}^{\top}{\mathbf{B}}^{\top}$. In the reversed order, as Theorem 19 says, ${\mathbf{B}}^{\top}{\mathbf{A}}^{\top} = \begin{bmatrix} 1 \cdot 1 + 1 \cdot 2 & 1 \cdot 0 + 1 \cdot 1 \\ 0 \cdot 1 + 1 \cdot 2 & 0 \cdot 0 + 1 \cdot 1 \end{bmatrix} = \begin{bmatrix} 3 & 1 \\ 2 & 1 \end{bmatrix}$, which matches ${(\mathbf{A}\mathbf{B})}^{\top}$.

Theorem 20 (A $1 \times 1$ product equals its own transpose)

Theorem 20 (A $1 \times 1$ product equals its own transpose) If $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}$ and $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}$ are vectors of length $p$ and $\mathbf{M}$ is a $p \times p$ matrix, then ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}$ is a number, and

\[ {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,{\mathbf{M}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}. \]

In particular, ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}$.

Proof

Proof. The product ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}$ is a $(1 \times p)(p \times p)(p \times 1)$ product, so it is a $1 \times 1$ matrix. The transpose of a $1 \times 1$ matrix is itself, so

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} &= {\mathopen{}\left({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\right)\mathclose{}}^{\top} && \text{(a } 1 \times 1 \text{ matrix equals its transpose)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,{\mathbf{M}}^{\top}\,{\mathopen{}\left({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\right)\mathclose{}}^{\top} && \text{(transpose of a product, twice)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,{\mathbf{M}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} && \text{(transposing twice changes nothing)} \end{aligned} \]

The transpose step is Theorem 19. Taking $\mathbf{M} = \mathbf{I}$ gives the last statement.

Example 25 (Two ways to write the same number)

Example 25 (Two ways to write the same number) Take $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} = (1, 2)$, $\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = (3, -1)$ and $\mathbf{M} = \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix}$. Then $\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = (1, -3)$, so

Also ${\mathbf{M}}^{\top} = \begin{bmatrix} 1 & 0 \\ 2 & 3 \end{bmatrix}$ and ${\mathbf{M}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} = (1, 8)$, so

\[ {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,{\mathbf{M}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} = 3 \cdot 1 + (-1) \cdot 8 = -5. \]

The two products are equal, as the theorem says. In least squares, this is why the two cross terms of ${(\mathbf{X}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}- \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})}^{\top}(\mathbf{X}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}- \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})$ are equal and can be combined.

Exercise 54 (Evaluating a product of a vector, a matrix and a vector)

Exercise 54 (Evaluating a product of a vector, a matrix and a vector) Theorem 20 turns ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,{\mathbf{M}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}$ back into ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}$, which can be the easier product of a vector, a matrix and a vector to compute. Let

\[ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} = \begin{bmatrix}1 \\ -1\end{bmatrix}, \qquad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = \begin{bmatrix}2 \\ 3\end{bmatrix}, \qquad \mathbf{M} = \begin{bmatrix}0 & 1 \\ 2 & 0\end{bmatrix}. \]

Find ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,{\mathbf{M}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}$.

Hint

By Theorem 20, it is enough to compute ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}$.

Solution 54

Solution 54. By Theorem 20, ${\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,{\mathbf{M}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}$. By Definition 16,

\[ \mathbf{M}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = \begin{bmatrix}0 \cdot 2 + 1 \cdot 3 \\ 2 \cdot 2 + 0 \cdot 3\end{bmatrix} = \begin{bmatrix}3 \\ 4\end{bmatrix}, \]

The answer is $-1$.

Definitions on this page

Definition 1 (Matrix)
Definition 2 (Equal matrices)
Definition 3 (Row of a matrix)
Definition 4 (Column of a matrix)
Definition 5 (Tensor)
Definition 6 (Matrix transpose)
Definition 7 (Zero matrix)
Definition 8 (Matrix addition)
Definition 9 (Scalar multiplication)
Definition 10 (Matrix multiplication)
Definition 11 (Square matrix)
Definition 12 (Main diagonal)
Definition 13 (Diagonal entry)
Definition 14 (Off-diagonal entry)
Definition 15 (Identity matrix)
Definition 16 (Matrix-vector multiplication)
Definition 17 (Linear system, system of linear equations)
Definition 18 (Additive function)
Definition 19 (Homogeneous function)
Definition 20 (Linear map, linear transformation, linear function)
Definition 21 (Linear operator)
Definition 22 (Matrix of a linear map)
Definition 23 (Translation)
Definition 24
Definition 25 (Offset of an affine map, intercept, bias)
Definition 26 (Augmented vector)
Definition 27 (Hyperplane)
Definition 28 (Normal vector of a hyperplane)

References

[1]

Wikipedia contributors, “Matrix (mathematics) — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Matrix_(mathematics)

[2]

Wikipedia contributors, “Tensor — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Tensor

[3]

Wikipedia contributors, “Transpose — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Transpose

[4]

Wikipedia contributors, “Zero matrix — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Zero_matrix

[5]

Wikipedia contributors, “Matrix addition — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Matrix_addition

[6]

Wikipedia contributors, “Scalar multiplication — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Scalar_multiplication

[7]

Wikipedia contributors, “Matrix multiplication — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Matrix_multiplication

[8]

Wikipedia contributors, “Square matrix — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Square_matrix

[9]

Wikipedia contributors, “Main diagonal — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Main_diagonal

[10]

Wikipedia contributors, “Identity matrix — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Identity_matrix

[11]

Wikipedia contributors, “System of linear equations — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/System_of_linear_equations

[12]

Wikipedia contributors, “Linear map — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Linear_map

[13]

Wikipedia contributors, “Translation (geometry) — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Translation_(geometry)

[14]

H. Zhou, “Matrices.” Lecture notes for Biostat 216, Mathematical Methods for Biostatistics, University of California, Los Angeles, 2024. Accessed: Oct. 05, 2026. [Online]. Available: https://ucla-biostat-216.github.io/2024fall/slides/03-matrix/03-matrix.html

[15]

S. Boyd and L. Vandenberghe, Introduction to applied linear algebra: Vectors, matrices, and least squares. Cambridge University Press, 2018. doi: 10.1017/9781108583664.

[16]

Wikipedia contributors, “Hyperplane — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Hyperplane

[17]

Wikipedia contributors, “Normal (geometry) — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Normal_(geometry)