Matrices
Definition 1 (Matrix) A matrix [1] of dimensions \(m \times n\) is a rectangular array of \(m \cdot n\) numbers, arranged in \(m\) rows and \(n\) columns:
\[ \mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix} \]
Remark 1 (Matrix entries, and vectors as matrices). The entry in row \(i\) and column \(j\) of \(\mathbf{A}\) is written \(a_{ij}\) or \((\mathbf{A})_{ij}\). For example, for the \(2 \times 3\) matrix
\[ \mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}, \]
\(a_{12} = 2\) and \(a_{23} = 6\).
A column vector of length \(p\) is a \(p \times 1\) matrix, and a row vector of length \(p\) is a \(1 \times p\) matrix. For example, the column vector \((7, 8)\) is a \(2 \times 1\) matrix, and the row vector \([7,\; 8]\) is a \(1 \times 2\) matrix.
Definition 2 (Equal matrices) Two matrices \(\mathbf{A}\) and \(\mathbf{B}\) are equal, written \(\mathbf{A} = \mathbf{B}\), if both are \(m \times n\) and \(a_{ij} = b_{ij}\) for every \(i = 1, \ldots, m\) and \(j = 1, \ldots, n\).
Example 1 (Equal and unequal matrices) \[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}, \qquad \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \ne \begin{bmatrix} 1 & 2 \\ 3 & 5 \end{bmatrix}, \qquad \begin{bmatrix} 1 & 2 & 3 & 4 \end{bmatrix} \ne \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}. \]
The second pair differs in one entry. The third pair lists the same four numbers in the same order, but one matrix is \(1 \times 4\) and the other \(2 \times 2\), so the two are not equal.
Exercise 1 (Solving for an unknown entry) Two matrices are equal only when every pair of matching entries agrees, so one matrix equation stands for several equations between numbers. Let
\[ \begin{bmatrix} 2 & x + 1 \\ 0 & 5 \end{bmatrix} = \begin{bmatrix} 2 & 4 \\ 0 & 5 \end{bmatrix}. \]
Find \(x\).
Both matrices are \(2 \times 2\), so by Definition 2 each entry on the left is the same number as the entry in the same position on the right.
Definition 3 (Row of a matrix) The \(i\)th row of an \(m \times n\) matrix \(\mathbf{A}\) is the \(1 \times n\) row vector of the entries in row \(i\):
\[ \mathbf{A}_{i,:} = \begin{bmatrix} a_{i1} & a_{i2} & \cdots & a_{in} \end{bmatrix}. \]
Definition 4 (Column of a matrix) The \(j\)th column of an \(m \times n\) matrix \(\mathbf{A}\) is the column vector of the entries in column \(j\):
\[ \mathbf{A}_{:,j} = \begin{bmatrix} a_{1j} \\ a_{2j} \\ \vdots \\ a_{mj} \end{bmatrix}. \]
Example 2 (Rows and columns of a matrix) For \(\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}\), the rows are \(\mathbf{A}_{1,:} = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix}\) and \(\mathbf{A}_{2,:} = \begin{bmatrix} 4 & 5 & 6 \end{bmatrix}\), and the columns are \(\mathbf{A}_{:,1} = \begin{bmatrix} 1 \\ 4 \end{bmatrix}\), \(\mathbf{A}_{:,2} = \begin{bmatrix} 2 \\ 5 \end{bmatrix}\) and \(\mathbf{A}_{:,3} = \begin{bmatrix} 3 \\ 6 \end{bmatrix}\).
Exercise 2 (Reading off a column of a matrix) When each row of a matrix records one observation, each column holds one measured quantity for every observation, so reading off a column extracts that quantity’s values. Let
\[ \mathbf{A} = \begin{bmatrix} 2 & 0 & -1 \\ 5 & 3 & 4 \end{bmatrix}. \]
Find \(\mathbf{A}_{:,2}\), the second column of \(\mathbf{A}\).
The \(j\)th column of an \(m \times n\) matrix lists the entries \(a_{1j}, \ldots, a_{mj}\) from top to bottom, as in Example 2.
Exercise 3 (Counting the rows and columns of a matrix) Data with one row per observation and one column per measured quantity is stored as a matrix, so the matrix’s number of rows is the number of observations and its number of columns is the number of quantities. Let
\[ \mathbf{C} = \begin{bmatrix} 4 & 0 \\ 1 & -3 \\ 2 & 7 \end{bmatrix}. \]
Find \(m\) and \(n\) such that \(\mathbf{C}\) is an \(m \times n\) matrix.
In Definition 1, \(m\) counts the rows and \(n\) counts the columns.
Exercise 4 (Reading off a row of a matrix) When each row of a matrix records one observation, reading off a row extracts every measured quantity for that one observation. Let
\[ \mathbf{D} = \begin{bmatrix} 1 & -4 & 2 \\ 0 & 6 & 3 \\ 5 & 5 & -1 \end{bmatrix}. \]
Find \(\mathbf{D}_{3,:}\), the third row of \(\mathbf{D}\).
A row of a matrix lists the entries \(a_{i1}, \ldots, a_{in}\) from left to right, as in Example 2.
Theorem 1 (A matrix is a column of its rows) An \(m \times n\) matrix \(\mathbf{A}\) is the column vector whose entries are its rows:
\[ \mathbf{A} = \begin{bmatrix} \mathbf{A}_{1,:} \\ \vdots \\ \mathbf{A}_{m,:} \end{bmatrix}. \]
Exercise 5 (Stacking rows into a matrix) Theorem 1 says that a matrix is determined by its rows, so data recorded one observation at a time can be assembled into a matrix by stacking those records. Let \(\mathbf{M}\) be the \(3 \times 2\) matrix with
\[ \mathbf{M}_{1,:} = \begin{bmatrix} 1 & 2 \end{bmatrix}, \qquad \mathbf{M}_{2,:} = \begin{bmatrix} 3 & -1 \end{bmatrix}, \qquad \mathbf{M}_{3,:} = \begin{bmatrix} 0 & 4 \end{bmatrix}. \]
What is \(\mathbf{M}_{:,2}\), the second column of \(\mathbf{M}\)?
Exercise 6 (Placing columns side by side) Theorem 2 says that a matrix is determined by its columns, so measurements recorded one quantity at a time can be assembled into a matrix by placing those lists side by side. Let \(\mathbf{N}\) be the \(2 \times 3\) matrix with
\[ \mathbf{N}_{:,1} = \begin{bmatrix} 2 \\ -1 \end{bmatrix}, \qquad \mathbf{N}_{:,2} = \begin{bmatrix} 0 \\ 3 \end{bmatrix}, \qquad \mathbf{N}_{:,3} = \begin{bmatrix} 4 \\ 1 \end{bmatrix}. \]
What is \(\mathbf{N}_{2,:}\), the second row of \(\mathbf{N}\)?
Definition 5 (Tensor) A tensor [2] of order \(k \ge 1\) is an array of numbers with \(k\) indices, \[ \mathcal{A} = \mathopen{}\left(a_{i_1 i_2 \cdots i_k}\right)\mathclose{}, \quad i_1 = 1, \ldots, n_1, \ \ldots, \ i_k = 1, \ldots, n_k, \] and its dimensions are \(n_1 \times n_2 \times \cdots \times n_k\).
Example 3 (Vectors, matrices and images as tensors) A vector of length \(p\) is a tensor of order \(1\), and an \(m \times n\) matrix is a tensor of order \(2\). A \(28 \times 28\) grayscale image is a matrix: the entry \(a_{ij}\) is the brightness of the pixel in row \(i\) and column \(j\). A \(28 \times 28\) color image stores three numbers per pixel (red, green, blue), so it is an order-3 tensor with dimensions \(28 \times 28 \times 3\), and \(a_{ijc}\) is the value of color channel \(c\) at the pixel in row \(i\) and column \(j\). It has \(28 \cdot 28 \cdot 3 = 2352\) entries.
Exercise 7 (Counting the entries of a batch of photos) Software for neural networks often stores a batch of color photos as one tensor, with one index for the photo, two for the pixel’s position, and one for the color channel, and the software stores one number per entry, so the count of entries determines how much memory the batch needs. Let \(\mathcal{B}\) be the order-4 tensor for a batch of \(10\) color photos of \(32 \times 32\) pixels, with \(n_1 = 10\), \(n_2 = 32\), \(n_3 = 32\) and \(n_4 = 3\).
How many entries does \(\mathcal{B}\) have?
Each index \(i_r\) in Definition 5 takes \(n_r\) values, and each choice of all four indices picks one entry, as in Example 3.
1 Matrix transpose
Definition 6 (Matrix transpose) The transpose [3] of an \(m \times n\) matrix \(\mathbf{A}\) is the \(n \times m\) matrix \({\mathbf{A}}^{\top}\) obtained by swapping the rows and columns of \(\mathbf{A}\):
\[({\mathbf{A}}^{\top})_{ij} = a_{ji}\]
Example 4 (Transposing a \(2 \times 3\) matrix) For \(\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}\),
\[ {\mathbf{A}}^{\top} = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix}, \]
a \(3 \times 2\) matrix. For instance, entry \((1, 2)\) of the transpose is \(a_{21} = 4\).
Exercise 8 (Transposing a \(3 \times 2\) matrix) The transpose turns the rows of a matrix into columns, just as the transpose of a vector turns a column into a row. Let
\[ \mathbf{B} = \begin{bmatrix} 1 & -2 \\ 0 & 3 \\ 4 & 5 \end{bmatrix}. \]
Compute \({\mathbf{B}}^{\top}\).
2 Matrix addition
Definition 7 (Zero matrix) The \(m \times n\) zero matrix [4] \(\mathbf{0}_{m \times n}\) (or \(\mathbf{0}\) when dimensions are clear from context) has all entries equal to zero:
\[ \mathbf{0}_{m \times n} = \begin{bmatrix} 0 & 0 & \cdots & 0 \\ 0 & 0 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 0 \end{bmatrix} \]
Example 5 (Zero matrices of two shapes) \[ \mathbf{0}_{2 \times 3} = \begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix}, \qquad \mathbf{0}_{3 \times 2} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \\ 0 & 0 \end{bmatrix}. \]
Both have all entries \(0\), but one is \(2 \times 3\) and the other \(3 \times 2\), so they are different matrices.
Exercise 9 (Whether a matrix of zeros is a given zero matrix) A zero matrix carries its shape in its subscript, because, as Example 5 shows, zero matrices of different shapes are different matrices. Let
\[ \mathbf{Z} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \\ 0 & 0 \end{bmatrix}. \]
Is \(\mathbf{Z} = \mathbf{0}_{2 \times 3}\)?
Compare the shapes of the two matrices, using Definition 2.
Definition 8 (Matrix addition) Two matrices \(\mathbf{A}\) and \(\mathbf{B}\) of the same dimensions \(m \times n\) can be added element-wise; their matrix sum [5] is:
\[(\mathbf{A} + \mathbf{B})_{ij} = a_{ij} + b_{ij}\]
Example 6 (Adding matrices, and a sum that is not defined) \[ \begin{aligned} \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} &= \begin{bmatrix} 1 + 0 & 2 + 1 \\ 3 + 1 & 4 + 0 \end{bmatrix} && \text{(add entry by entry)} \\ &= \begin{bmatrix} 1 & 3 \\ 4 & 4 \end{bmatrix}. && \text{(add)} \end{aligned} \]
A \(2 \times 2\) matrix and a \(2 \times 3\) matrix cannot be added: they do not have the same dimensions, so entry \((1, 3)\) of the sum would have no first term.
Exercise 10 (Adding two \(2 \times 2\) matrices) A matrix sum combines two tables of the same shape entry by entry, such as two months of counts recorded in the same rows and columns. Let
\[ \mathbf{A} = \begin{bmatrix} 1 & -1 \\ 2 & 0 \end{bmatrix}, \qquad \mathbf{B} = \begin{bmatrix} 3 & 2 \\ -2 & 5 \end{bmatrix}. \]
Compute \(\mathbf{A} + \mathbf{B}\).
Add the entries in the same position, as in Example 6.
Theorem 3 (Matrix addition is commutative) For \(m \times n\) matrices \(\mathbf{A}\) and \(\mathbf{B}\), matrix addition is commutative:
\[\mathbf{A} + \mathbf{B} = \mathbf{B} + \mathbf{A}\]
Exercise 11 (Proving that matrix addition is commutative) Theorem 3 lets two matrices be added in either order, just as two numbers can be (sums are symmetric).
Prove Theorem 3.
Two \(m \times n\) matrices are equal when their \((i, j)\) entries agree for every \(i\) and \(j\). Express the \((i, j)\) entry of each side with Definition 8.
Theorem 4 (Matrix addition is associative) For \(m \times n\) matrices \(\mathbf{A}\), \(\mathbf{B}\), and \(\mathbf{C}\), matrix addition is associative:
\[(\mathbf{A} + \mathbf{B}) + \mathbf{C} = \mathbf{A} + (\mathbf{B} + \mathbf{C})\]
Exercise 12 (Proving that matrix addition is associative) Theorem 4 lets three matrices be added without parentheses, just as three numbers can be (sums are associative).
Prove Theorem 4.
Two \(m \times n\) matrices are equal when their \((i, j)\) entries agree for every \(i\) and \(j\). Apply Definition 8 twice on each side.
Theorem 5 (Zero matrix is the additive identity) For an \(m \times n\) matrix \(\mathbf{A}\):
\[\mathbf{A} + \mathbf{0}_{m \times n} = \mathbf{A}\]
Exercise 13 (Proving that the zero matrix is the additive identity) Theorem 5 says that adding the zero matrix leaves a matrix unchanged, just as adding the number \(0\) leaves a number unchanged.
Prove Theorem 5.
Express the \((i, j)\) entry of \(\mathbf{A} + \mathbf{0}_{m \times n}\) with Definition 8, and use that every entry of \(\mathbf{0}_{m \times n}\) is \(0\).
Theorem 6 (Additive inverse) For any \(m \times n\) matrix \(\mathbf{A}\), the \(m \times n\) matrix \(-\mathbf{A}\) (defined by \((-\mathbf{A})_{ij} = -a_{ij}\)) satisfies:
\[\mathbf{A} + (-\mathbf{A}) = \mathbf{0}_{m \times n}\]
Exercise 14 (Proving that a matrix plus its negation is zero) Theorem 6 says that adding \(-\mathbf{A}\) to a matrix \(\mathbf{A}\) gives the zero matrix, just as adding \(-a\) to a number \(a\) gives \(0\). Here \(-\mathbf{A}\), the negation of \(\mathbf{A}\), negates every entry of \(\mathbf{A}\).
Prove Theorem 6.
Express the \((i, j)\) entry of \(\mathbf{A} + (-\mathbf{A})\) with Definition 8, and use \((-\mathbf{A})_{ij} = -a_{ij}\) from Theorem 6.
3 Scalar multiplication
Definition 9 (Scalar multiplication) The scalar multiple [6] of a matrix \(\mathbf{A}\) by a scalar \(c\) is:
\[(c\mathbf{A})_{ij} = c \cdot a_{ij}\]
Example 7 (A scalar multiple) \[ \begin{aligned} 3 \begin{bmatrix} 1 & -2 \\ 0 & 4 \end{bmatrix} &= \begin{bmatrix} 3 \cdot 1 & 3 \cdot(-2) \\ 3 \cdot 0 & 3 \cdot 4 \end{bmatrix} \\ &= \begin{bmatrix} 3 & -6 \\ 0 & 12 \end{bmatrix}. \end{aligned} \]
Exercise 15 (Multiplying a matrix by a negative factor) A scalar multiple rescales every entry of a matrix by the same factor, as when a table of distances is converted from meters to centimeters. A negative factor flips the sign of every entry, and a factor between \(-1\) and \(1\) shrinks every entry toward \(0\). Let
\[ \mathbf{C} = \begin{bmatrix} 4 & -2 \\ 0 & 6 \end{bmatrix}. \]
Compute \(-\frac{1}{2} \mathbf{C}\).
Multiply each entry by \(-\frac{1}{2}\), as in Example 7.