Skip to main content

📋 Other Formats: 🎞️ Slides

Special Matrices and Decompositions

Published

Last modified: 2026-11-20 09:01:42 (PST)

1 Special Matrices

Some special matrices appeared earlier:

DefDefinition 1 (Order of a square matrix)

Definition 1 (Order of a square matrix) The order of a square matrix (Definition 3 in Matrix Multiplication) is its number of rows, which equals its number of columns.

ExmExample 1 (Orders of square matrices)

Example 1 (Orders of square matrices) \(\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}\) has order \(2\), and \([5]\) has order \(1\). A \(2 \times 3\) matrix has no order, since it is not square (Example 5 in Matrix Multiplication).

DefDefinition 2 (Matrix power)

Definition 2 (Matrix power) For a square matrix \(\mathbf{A}\) of order \(p\) and a positive integer \(k\), the \(k\)-th power of \(\mathbf{A}\) is:

\[\mathbf{A}^k = \underbrace{\mathbf{A}\,\mathbf{A}\cdots\mathbf{A}}_{k \text{ copies}}\]

In particular, \(\mathbf{A}^2 = \mathbf{A}\mathbf{A}\).

ExmExample 2 (Powers of a \(2 \times 2\) matrix)

Example 2 (Powers of a \(2 \times 2\) matrix) Let \(\mathbf{A} = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}\). Then

\[ \begin{aligned} \mathbf{A}^2 &= \begin{bmatrix} 1 \cdot 1 + 1 \cdot 0 & 1 \cdot 1 + 1 \cdot 1 \\ 0 \cdot 1 + 1 \cdot 0 & 0 \cdot 1 + 1 \cdot 1 \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \begin{bmatrix} 1 & 2 \\ 0 & 1 \end{bmatrix}, && \text{(multiply and add)} \\ \mathbf{A}^3 = \mathbf{A}^2 \mathbf{A} &= \begin{bmatrix} 1 \cdot 1 + 2 \cdot 0 & 1 \cdot 1 + 2 \cdot 1 \\ 0 \cdot 1 + 1 \cdot 0 & 0 \cdot 1 + 1 \cdot 1 \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \begin{bmatrix} 1 & 3 \\ 0 & 1 \end{bmatrix}. && \text{(multiply and add)} \end{aligned} \]

A \(2 \times 3\) matrix \(\mathbf{B}\) is not square (Definition 3 in Matrix Multiplication), so \(\mathbf{B}^2\) is not defined; indeed \(\mathbf{B} \mathbf{B}\) would multiply a matrix with \(3\) columns by one with \(2\) rows, which Definition 1 in Matrix Multiplication does not allow.

DefDefinition 3 (Symmetric matrix)

Definition 3 (Symmetric matrix) A square matrix \(\mathbf{A}\) is symmetric [1] if \({\mathbf{A}}^{\top} = \mathbf{A}\), i.e., \(a_{ij} = a_{ji}\) for all \(i\) and \(j\).

RemarkRemark 1 (Symmetric matrices in statistics)

Remark 1 (Symmetric matrices in statistics). Covariance matrices are symmetric: entry \((i, j)\) of the covariance matrix of a random vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{Y}}\vphantom{Y}}\) is \({\operatorname{Cov}\mathopen{}\left(Y_i, Y_j\right)\mathclose{}}\), and \({\operatorname{Cov}\mathopen{}\left(Y_i, Y_j\right)\mathclose{}} = {\operatorname{Cov}\mathopen{}\left(Y_j, Y_i\right)\mathclose{}}\). For example, if \({\operatorname{Var}\mathopen{}\left(Y_1\right)\mathclose{}} = 4\), \({\operatorname{Var}\mathopen{}\left(Y_2\right)\mathclose{}} = 9\), and \({\operatorname{Cov}\mathopen{}\left(Y_1, Y_2\right)\mathclose{}} = 1\), the covariance matrix of \((Y_1, Y_2)\) is

\[ \begin{bmatrix} 4 & 1 \\ 1 & 9 \end{bmatrix}. \]

ExmExample 3 (A matrix that is not symmetric)

Example 3 (A matrix that is not symmetric) \(\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}\) is not symmetric:

\[ \begin{aligned} a_{12} &= 2 \\ &\ne 3 \\ &= a_{21}. \end{aligned} \]

DefDefinition 4 (Diagonal matrix)

Definition 4 (Diagonal matrix) A square matrix \(\mathbf{D}\) is a diagonal matrix [2] if all off-diagonal entries are zero: \(d_{ij} = 0\) whenever \(i \neq j\):

\[ \mathbf{D} = \begin{bmatrix} d_1 & 0 & \cdots & 0 \\ 0 & d_2 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & d_p \end{bmatrix} \]

RemarkRemark 2 (\(\operatorname{diag}\) notation)

Remark 2 (\(\operatorname{diag}\) notation). For numbers \(d_1, \ldots, d_p\), \(\operatorname{diag}(d_1, \ldots, d_p)\) is the \(p \times p\) diagonal matrix with diagonal entries \(d_1, \ldots, d_p\). For example,

\[ \operatorname{diag}(2, 5) = \begin{bmatrix} 2 & 0 \\ 0 & 5 \end{bmatrix}, \]

and the \(p \times p\) identity matrix is \(\mathbf{I}_p = \operatorname{diag}(1, \ldots, 1)\).

ExmExample 4 (A matrix that is not diagonal)

Example 4 (A matrix that is not diagonal) \(\mathbf{D} = \begin{bmatrix} 2 & 1 \\ 0 & 5 \end{bmatrix}\) is not diagonal: its off-diagonal entry \(d_{12} = 1 \ne 0\).

DefDefinition 5 (Matrix inverse)

Definition 5 (Matrix inverse) For a square \(p \times p\) matrix \(\mathbf{A}\), the inverse \(\mathbf{A}^{-1}\) (if it exists) is the unique matrix satisfying:

\[ \begin{aligned} \mathbf{A}\,\mathbf{A}^{-1} &= \mathbf{A}^{-1}\,\mathbf{A} \\ &= \mathbf{I}_p \end{aligned} \]

DefDefinition 6 (Invertible matrix, non-singular matrix)

Definition 6 (Invertible matrix, non-singular matrix) A \(p \times p\) matrix \(\mathbf{A}\) is invertible [3] (or non-singular) if some \(p \times p\) matrix \(\mathbf{B}\) satisfies

\[ \begin{aligned} \mathbf{A}\mathbf{B} &= \mathbf{B}\mathbf{A} \\ &= \mathbf{I}_p, \end{aligned} \]

where \(\mathbf{I}_p\) is the identity matrix (Definition 7 in Matrix Multiplication). A square matrix that is not invertible is singular.

ExmExample 5 (An invertible matrix and a singular one)

Example 5 (An invertible matrix and a singular one) The matrix \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 0 & 1 \end{bmatrix}\) is invertible, with \(\mathbf{A}^{-1} = \begin{bmatrix} 0.5 & -0.5 \\ 0 & 1 \end{bmatrix}\): multiplying out gives

\[ \begin{aligned} \mathbf{A}\mathbf{A}^{-1} &= \mathbf{A}^{-1}\mathbf{A} \\ &= \mathbf{I}_2. \end{aligned} \]

The matrix \(\mathbf{M} = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}\) is singular: for any \(2 \times 2\) matrix \(\mathbf{C}\), the two rows of \(\mathbf{M}\mathbf{C}\) are equal, so \(\mathbf{M}\mathbf{C}\) can never be \(\mathbf{I}_2\), whose two rows differ.

RemarkRemark 3 (Invertible matrices and inverses)

Remark 3 (Invertible matrices and inverses). Let \(\mathbf{A}\) be a \(p \times p\) matrix, and suppose that two \(p \times p\) matrices \(\mathbf{B}\) and \(\mathbf{C}\) satisfy

\[ \begin{aligned} \mathbf{A}\mathbf{B} &= \mathbf{B}\mathbf{A} \\ &= \mathbf{I}_p \end{aligned} \]

and

\[ \begin{aligned} \mathbf{A}\mathbf{C} &= \mathbf{C}\mathbf{A} \\ &= \mathbf{I}_p. \end{aligned} \]

Then

\[ \begin{aligned} \mathbf{B} &= \mathbf{B}\,\mathbf{I}_p && \text{(identity matrix)} \\ &= \mathbf{B}(\mathbf{A}\mathbf{C}) && \text{(} \mathbf{A}\mathbf{C} = \mathbf{I}_p \text{)} \\ &= (\mathbf{B}\mathbf{A})\mathbf{C} && \text{(matrix multiplication is associative)} \\ &= \mathbf{I}_p\,\mathbf{C} && \text{(} \mathbf{B}\mathbf{A} = \mathbf{I}_p \text{)} \\ &= \mathbf{C} && \text{(identity matrix)} \end{aligned} \]

The steps use Theorem 1 in Matrix Multiplication and Theorem 4 in Matrix Multiplication. Since \(\mathbf{B} = \mathbf{C}\), at most one matrix satisfies these equations, which is the uniqueness that Definition 5 asserts. When \(\mathbf{A}\) is invertible, the matrix \(\mathbf{B}\) in Definition 6 is therefore the inverse \(\mathbf{A}^{-1}\) of \(\mathbf{A}\), and \(\mathbf{A}\) is invertible exactly when it has an inverse. For example, for \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 0 & 1 \end{bmatrix}\) in Example 5, \(\begin{bmatrix} 0.5 & -0.5 \\ 0 & 1 \end{bmatrix}\) is the only \(2 \times 2\) matrix \(\mathbf{B}\) with

\[ \begin{aligned} \mathbf{A}\mathbf{B} &= \mathbf{B}\mathbf{A} \\ &= \mathbf{I}_2, \end{aligned} \]

so it is \(\mathbf{A}^{-1}\).

LemTheorem 1 (Inverse of a product)

Theorem 1 (Inverse of a product) If \(\mathbf{A}\) and \(\mathbf{B}\) are invertible \(p \times p\) matrices, then \(\mathbf{A}\mathbf{B}\) is invertible, and

\[(\mathbf{A}\mathbf{B})^{-1} = \mathbf{B}^{-1}\mathbf{A}^{-1}\]

Proof. Multiply \(\mathbf{A}\mathbf{B}\) by the candidate inverse on the right:

\[ \begin{aligned} (\mathbf{A}\mathbf{B})(\mathbf{B}^{-1}\mathbf{A}^{-1}) &= \mathbf{A}(\mathbf{B}\mathbf{B}^{-1})\mathbf{A}^{-1} && \text{(matrix multiplication is associative)} \\ &= \mathbf{A}\,\mathbf{I}_p\,\mathbf{A}^{-1} && \text{(definition of } \mathbf{B}^{-1} \text{)} \\ &= \mathbf{A}\mathbf{A}^{-1} && \text{(identity matrix)} \\ &= \mathbf{I}_p && \text{(definition of } \mathbf{A}^{-1} \text{)} \end{aligned} \]

and on the left:

\[ \begin{aligned} (\mathbf{B}^{-1}\mathbf{A}^{-1})(\mathbf{A}\mathbf{B}) &= \mathbf{B}^{-1}(\mathbf{A}^{-1}\mathbf{A})\mathbf{B} && \text{(matrix multiplication is associative)} \\ &= \mathbf{B}^{-1}\,\mathbf{I}_p\,\mathbf{B} && \text{(definition of } \mathbf{A}^{-1} \text{)} \\ &= \mathbf{B}^{-1}\mathbf{B} && \text{(identity matrix)} \\ &= \mathbf{I}_p && \text{(definition of } \mathbf{B}^{-1} \text{)} \end{aligned} \]

So \(\mathbf{B}^{-1}\mathbf{A}^{-1}\) satisfies Definition 5 for \(\mathbf{A}\mathbf{B}\). The steps use Theorem 1 in Matrix Multiplication and Theorem 4 in Matrix Multiplication.

LemTheorem 2 (Inverse of a transpose)

Theorem 2 (Inverse of a transpose) If \(\mathbf{A}\) is an invertible \(p \times p\) matrix, then \({\mathbf{A}}^{\top}\) is invertible, and

\[{\mathopen{}\left({\mathbf{A}}^{\top}\right)\mathclose{}}^{-1} = {\mathopen{}\left(\mathbf{A}^{-1}\right)\mathclose{}}^{\top}\]

Proof. Multiply \({\mathbf{A}}^{\top}\) by the candidate inverse on each side:

\[ \begin{aligned} {\mathbf{A}}^{\top}\,{\mathopen{}\left(\mathbf{A}^{-1}\right)\mathclose{}}^{\top} &= {\mathopen{}\left(\mathbf{A}^{-1}\mathbf{A}\right)\mathclose{}}^{\top} && \text{(transpose of a product)} \\ &= {\mathbf{I}_p}^{\top} && \text{(definition of } \mathbf{A}^{-1} \text{)} \\ &= \mathbf{I}_p && \text{(} \mathbf{I}_p \text{ is symmetric)} \end{aligned} \]

\[ \begin{aligned} {\mathopen{}\left(\mathbf{A}^{-1}\right)\mathclose{}}^{\top}\,{\mathbf{A}}^{\top} &= {\mathopen{}\left(\mathbf{A}\mathbf{A}^{-1}\right)\mathclose{}}^{\top} && \text{(transpose of a product)} \\ &= {\mathbf{I}_p}^{\top} && \text{(definition of } \mathbf{A}^{-1} \text{)} \\ &= \mathbf{I}_p && \text{(} \mathbf{I}_p \text{ is symmetric)} \end{aligned} \]

So \({\mathopen{}\left(\mathbf{A}^{-1}\right)\mathclose{}}^{\top}\) satisfies Definition 5 for \({\mathbf{A}}^{\top}\). The first step of each display is Theorem 6 in Matrix Multiplication.

ExmExample 6 (Inverting a transpose)

Example 6 (Inverting a transpose) For \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 0 & 1 \end{bmatrix}\), Example 5 gives \(\mathbf{A}^{-1} = \begin{bmatrix} 0.5 & -0.5 \\ 0 & 1 \end{bmatrix}\), so Theorem 2 says

\[ \begin{aligned} {\mathopen{}\left({\mathbf{A}}^{\top}\right)\mathclose{}}^{-1} &= {\mathopen{}\left(\begin{bmatrix} 2 & 0 \\ 1 & 1 \end{bmatrix}\right)\mathclose{}}^{-1} \\ &= {\mathopen{}\left(\mathbf{A}^{-1}\right)\mathclose{}}^{\top} \\ &= \begin{bmatrix} 0.5 & 0 \\ -0.5 & 1 \end{bmatrix}, \end{aligned} \]

and multiplying \(\begin{bmatrix} 2 & 0 \\ 1 & 1 \end{bmatrix}\begin{bmatrix} 0.5 & 0 \\ -0.5 & 1 \end{bmatrix}\) out does give \(\mathbf{I}_2\).

CorCorollary 1 (The inverse of a symmetric matrix is symmetric)

Corollary 1 (The inverse of a symmetric matrix is symmetric) If \(\mathbf{A}\) is symmetric and invertible, then \(\mathbf{A}^{-1}\) is symmetric.

Proof. \[ \begin{aligned} {\mathopen{}\left(\mathbf{A}^{-1}\right)\mathclose{}}^{\top} &= {\mathopen{}\left({\mathbf{A}}^{\top}\right)\mathclose{}}^{-1} && \text{(inverse of a transpose)} \\ &= \mathbf{A}^{-1} && \text{(} \mathbf{A} \text{ is symmetric)} \end{aligned} \]

The first step is Theorem 2.

ExmExample 7 (Inverting a symmetric matrix)

Example 7 (Inverting a symmetric matrix) \(\mathbf{S} = \begin{bmatrix} 2 & 1 \\ 1 & 1 \end{bmatrix}\) is symmetric, and \(\mathbf{S}^{-1} = \begin{bmatrix} 1 & -1 \\ -1 & 2 \end{bmatrix}\) is symmetric too, as Corollary 1 says.

LemTheorem 3 (An invertible factor can be canceled)

Theorem 3 (An invertible factor can be canceled) Let \(\mathbf{A}\) be an invertible \(p \times p\) matrix (Definition 6), and let \(\mathbf{B}\) and \(\mathbf{C}\) be \(p \times n\) matrices. If \(\mathbf{A}\mathbf{B} = \mathbf{A}\mathbf{C}\), then \(\mathbf{B} = \mathbf{C}\).

Proof. \[ \begin{aligned} \mathbf{B} &= \mathbf{I}_p\,\mathbf{B} && \text{(}\href{linear-algebra-matrix-multiplication.qmd#thm-identity}{\text{Theorem~4 in Matrix Multiplication}}\text{)} \\ &= (\mathbf{A}^{-1}\mathbf{A})\,\mathbf{B} && \text{(}\href{#def-matrix-inverse}{\text{Definition~5}}\text{)} \\ &= \mathbf{A}^{-1}(\mathbf{A}\mathbf{B}) && \text{(}\href{linear-algebra-matrix-multiplication.qmd#thm-matmul-assoc}{\text{Theorem~1 in Matrix Multiplication}}\text{)} \\ &= \mathbf{A}^{-1}(\mathbf{A}\mathbf{C}) && \text{(} \mathbf{A}\mathbf{B} = \mathbf{A}\mathbf{C} \text{)} \\ &= (\mathbf{A}^{-1}\mathbf{A})\,\mathbf{C} && \text{(}\href{linear-algebra-matrix-multiplication.qmd#thm-matmul-assoc}{\text{Theorem~1 in Matrix Multiplication}}\text{)} \\ &= \mathbf{I}_p\,\mathbf{C} && \text{(}\href{#def-matrix-inverse}{\text{Definition~5}}\text{)} \\ &= \mathbf{C}. && \text{(}\href{linear-algebra-matrix-multiplication.qmd#thm-identity}{\text{Theorem~4 in Matrix Multiplication}}\text{)} \end{aligned} \]

\(\square\)

CorCorollary 2 (An invertible factor can be canceled on the right)

Corollary 2 (An invertible factor can be canceled on the right) Let \(\mathbf{A}\) be an invertible \(p \times p\) matrix, and let \(\mathbf{B}\) and \(\mathbf{C}\) be \(n \times p\) matrices. If \(\mathbf{B}\mathbf{A} = \mathbf{C}\mathbf{A}\), then \(\mathbf{B} = \mathbf{C}\).

The proof of Theorem 3 works with every product reversed: multiply both sides on the right by \(\mathbf{A}^{-1}\).

ExmExample 8 (Solving for a matrix by canceling an invertible factor)

Example 8 (Solving for a matrix by canceling an invertible factor) Let \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 0 & 1 \end{bmatrix}\), which is invertible with \(\mathbf{A}^{-1} = \begin{bmatrix} 0.5 & -0.5 \\ 0 & 1 \end{bmatrix}\) (Example 5). If \(\mathbf{A}\mathbf{B} = \begin{bmatrix} 3 \\ 1 \end{bmatrix}\) for a \(2 \times 1\) matrix \(\mathbf{B}\), then multiplying both sides on the left by \(\mathbf{A}^{-1}\), as in the proof of Theorem 3,

\[ \begin{aligned} \mathbf{B} &= \mathbf{A}^{-1} \begin{bmatrix} 3 \\ 1 \end{bmatrix} \\ &= \begin{bmatrix} 1 \\ 1 \end{bmatrix}. \end{aligned} \]

Check: \(\mathbf{A} \begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 2 + 1 \\ 0 + 1 \end{bmatrix} = \begin{bmatrix} 3 \\ 1 \end{bmatrix}\).

ExmExample 9 (A product of two nonzero matrices can be the zero matrix)

Example 9 (A product of two nonzero matrices can be the zero matrix) With

\[ \mathbf{A} = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}, \qquad \mathbf{B} = \begin{bmatrix} 0 & 0 \\ 0 & 1 \end{bmatrix}, \]

neither matrix is the zero matrix (Definition 7 in Matrices), but

\[ \begin{aligned} \mathbf{A}\mathbf{B} &= \begin{bmatrix} 1 \cdot 0 + 0 \cdot 0 & 1 \cdot 0 + 0 \cdot 1 \\ 0 \cdot 0 + 0 \cdot 0 & 0 \cdot 0 + 0 \cdot 1 \end{bmatrix} \\ &= \mathbf{0}_{2 \times 2}. \end{aligned} \]

For numbers, \(ab = 0\) forces \(a = 0\) or \(b = 0\). For matrices it does not.

Because \(\mathbf{A}\mathbf{0}_{2 \times 2} = \mathbf{0}_{2 \times 2} = \mathbf{A}\mathbf{B}\) while \(\mathbf{B} \neq \mathbf{0}_{2 \times 2}\), the factor \(\mathbf{A}\) cannot be canceled from \(\mathbf{A}\mathbf{B} = \mathbf{A}\mathbf{0}_{2 \times 2}\). This does not contradict Theorem 3, because \(\mathbf{A}\) is singular: its second row is zero, so \(\mathbf{A}\mathbf{C}\) has a zero second row for every \(2 \times 2\) matrix \(\mathbf{C}\), and \(\mathbf{A}\mathbf{C}\) can never be \(\mathbf{I}_2\).

RemarkRemark 4 (Matrix arithmetic compared with scalar arithmetic)

Remark 4 (Matrix arithmetic compared with scalar arithmetic). Sums and differences. Matrix addition has every property that addition of numbers has: it is commutative (Theorem 3 in Matrices), associative (Theorem 4 in Matrices), has the zero matrix as its identity (Theorem 5 in Matrices), and gives each matrix a negative (Theorem 6 in Matrices). The only new condition is that \(\mathbf{A} + \mathbf{B}\) is defined only when \(\mathbf{A}\) and \(\mathbf{B}\) have the same dimensions.

Products. Matrix multiplication keeps three properties of multiplication of numbers: it is associative (Theorem 1 in Matrix Multiplication), it is distributive over addition (Theorem 2 in Matrix Multiplication), and it has an identity, \(\mathbf{I}_p\) (Theorem 4 in Matrix Multiplication). It differs from multiplication of numbers in four ways:

  • \(\mathbf{A}\mathbf{B}\) is defined only when the number of columns of \(\mathbf{A}\) equals the number of rows of \(\mathbf{B}\) (Definition 1 in Matrix Multiplication).
  • It is not commutative: \(\mathbf{A}\mathbf{B} \neq \mathbf{B}\mathbf{A}\) in general (Remark 1 in Matrix Multiplication).
  • A nonzero square matrix need not have an inverse (Example 5).
  • \(\mathbf{A}\mathbf{B} = \mathbf{0}\) can hold with \(\mathbf{A} \neq \mathbf{0}\) and \(\mathbf{B} \neq \mathbf{0}\) (Example 9), so a factor cannot be canceled from \(\mathbf{A}\mathbf{B} = \mathbf{A}\mathbf{C}\) unless it is invertible (Theorem 3).
DefDefinition 7 (Idempotent matrix)

Definition 7 (Idempotent matrix) A square matrix \(\mathbf{A}\) is idempotent [4] if

\[\mathbf{A}^2 = \mathbf{A}\]

ExmExample 10 (An idempotent matrix)

Example 10 (An idempotent matrix) For \(\mathbf{P} = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}\),

\[ \begin{aligned} \mathbf{P}^2 &= \begin{bmatrix} 1 \cdot 1 + 0 \cdot 0 & 1 \cdot 0 + 0 \cdot 0 \\ 0 \cdot 1 + 0 \cdot 0 & 0 \cdot 0 + 0 \cdot 0 \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix} \\ &= \mathbf{P}, && \text{(multiply and add)} \end{aligned} \]

so \(\mathbf{P}\) is idempotent.

ExmExample 11 (A matrix that is not idempotent)

Example 11 (A matrix that is not idempotent) For \(\mathbf{A} = \begin{bmatrix} 2 & 0 \\ 0 & 0 \end{bmatrix}\),

\[ \begin{aligned} \mathbf{A}^2 &= \begin{bmatrix} 2 \cdot 2 + 0 \cdot 0 & 2 \cdot 0 + 0 \cdot 0 \\ 0 \cdot 2 + 0 \cdot 0 & 0 \cdot 0 + 0 \cdot 0 \end{bmatrix} \\ &= \begin{bmatrix} 4 & 0 \\ 0 & 0 \end{bmatrix} \\ &\ne \mathbf{A}, \end{aligned} \]

so \(\mathbf{A}\) is not idempotent.

DefDefinition 8 (Orthogonal projection matrix)

Definition 8 (Orthogonal projection matrix) A square matrix \(\mathbf{P}\) is an orthogonal projection matrix (also called an orthogonal projector) if it is both symmetric (Definition 3) and idempotent (Definition 7):

\[{\mathbf{P}}^{\top} = \mathbf{P} \qquad \text{and} \qquad \mathbf{P}^2 = \mathbf{P}\]

DefDefinition 9 (Oblique projection)

Definition 9 (Oblique projection) A square matrix is an oblique projection if it is idempotent (Definition 7) but not symmetric (Definition 3).

ExmExample 12 (An orthogonal projection and an oblique one)

Example 12 (An orthogonal projection and an oblique one) \(\mathbf{P} = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}\) is symmetric, and \(\mathbf{P}^2 = \mathbf{P}\), so \(\mathbf{P}\) is an orthogonal projection matrix; it maps \((v_1, v_2)\) to \((v_1, 0)\).

\(\mathbf{Q} = \begin{bmatrix} 1 & 1 \\ 0 & 0 \end{bmatrix}\) is idempotent:

\[ \begin{aligned} \mathbf{Q}^2 &= \begin{bmatrix} 1 \cdot 1 + 1 \cdot 0 & 1 \cdot 1 + 1 \cdot 0 \\ 0 \cdot 1 + 0 \cdot 0 & 0 \cdot 1 + 0 \cdot 0 \end{bmatrix} \\ &= \begin{bmatrix} 1 & 1 \\ 0 & 0 \end{bmatrix} \\ &= \mathbf{Q}, \end{aligned} \]

but \({\mathbf{Q}}^{\top} \neq \mathbf{Q}\), so \(\mathbf{Q}\) is an oblique projection, not an orthogonal projection matrix.

RemarkRemark 5 (What “projection matrix” means in these notes)

Remark 5 (What “projection matrix” means in these notes). Some texts call any idempotent matrix a projection matrix, so that both \(\mathbf{P}\) and \(\mathbf{Q}\) in Example 12 would count as one. Regression texts often say “projection matrix” when they mean an orthogonal one. In these notes, “projection matrix” always means an orthogonal projection matrix in the sense of Definition 8, such as \(\mathbf{P}\) in Example 12, and never an oblique projection (Definition 9) such as \(\mathbf{Q}\).

LemTheorem 4 (Complement of a projection matrix)

Theorem 4 (Complement of a projection matrix) If \(\mathbf{P}\) is a \(p \times p\) orthogonal projection matrix (Definition 8), then \(\mathbf{I}_p - \mathbf{P}\) is also an orthogonal projection matrix.

Proof. We verify symmetry and idempotency.

Symmetry: \[ \begin{aligned} {(\mathbf{I} - \mathbf{P})}^{\top} &= {\mathbf{I}}^{\top} - {\mathbf{P}}^{\top} \\ &= \mathbf{I} - \mathbf{P} \end{aligned} \]

Idempotency: \[\begin{aligned} (\mathbf{I} - \mathbf{P})^2 &= (\mathbf{I} - \mathbf{P})(\mathbf{I} - \mathbf{P}) \\ &= \mathbf{I} - \mathbf{P} - \mathbf{P} + \mathbf{P}^2 \\ &= \mathbf{I} - \mathbf{P} - \mathbf{P} + \mathbf{P} \\ &= \mathbf{I} - \mathbf{P} \end{aligned}\]

LemTheorem 5 (Projection matrices produce orthogonal decompositions)

Theorem 5 (Projection matrices produce orthogonal decompositions) If \(\mathbf{P}\) is a \(p \times p\) orthogonal projection matrix (Definition 8) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) is any vector of length \(p\), then the two components of the decomposition

\[\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = \underbrace{\mathbf{P}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}_{\text{projected}} + \underbrace{(\mathbf{I}_p - \mathbf{P})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}_{\text{residual}}\]

are orthogonal:

\[\mathbf{P}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \;\perp\; (\mathbf{I}_p - \mathbf{P})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\]

Proof. \[\begin{aligned} {(\mathbf{P}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}})}^{\top}\,(\mathbf{I} - \mathbf{P})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,{\mathbf{P}}^{\top}\,(\mathbf{I} - \mathbf{P})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,\mathbf{P}\,(\mathbf{I} - \mathbf{P})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,(\mathbf{P} - \mathbf{P}^2)\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,(\mathbf{P} - \mathbf{P})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\,\mathbf{0}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \\ &= 0 \end{aligned}\]

where the second line uses symmetry (\({\mathbf{P}}^{\top} = \mathbf{P}\)) and the fourth line uses idempotency (\(\mathbf{P}^2 = \mathbf{P}\)).

LemTheorem 6 (The diagonal of an orthogonal projection matrix lies between 0 and 1)

Theorem 6 (The diagonal of an orthogonal projection matrix lies between 0 and 1) Let \(\mathbf{P}\) be an \(n \times n\) orthogonal projection matrix with entries \(p_{ij}\). Then

\[ 0 \le p_{ii} \le 1 \qquad \text{for every } i = 1, \dots, n. \]

Proof. Since \(\mathbf{P}\) is idempotent and symmetric, \(p_{ij} = p_{ji}\) and \(\mathbf{P}^2 = \mathbf{P}\). Compare the \((i, i)\) entries of \(\mathbf{P}^2\) and \(\mathbf{P}\):

\[ \begin{aligned} p_{ii} &= (\mathbf{P}^2)_{ii} && \text{(idempotent)} \\ &= \sum_{j=1}^{n} p_{ij}\, p_{ji} && \text{(definition of matrix multiplication)} \\ &= \sum_{j=1}^{n} p_{ij}^2 && \text{(symmetric)} \\ &= p_{ii}^2 + \sum_{j \ne i} p_{ij}^2. && \text{(separate the term } j = i \text{)} \end{aligned} \]

Each \(p_{ij}^2\) is non-negative, so \(p_{ii} \ge 0\). The same equation also gives \(p_{ii} \ge p_{ii}^2\), so \(p_{ii}(1 - p_{ii}) \ge 0\). Since \(p_{ii} \ge 0\), this requires \(p_{ii} \le 1\).

ExmExample 13 (Diagonal of a projection onto a line)

Example 13 (Diagonal of a projection onto a line) Let \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} = (1, 1, 0)\). The matrix \(\mathbf{P} = \frac{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}}{{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}\) is the orthogonal projection onto the line through \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}\). To see this, take the unit vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{q}}\vphantom{q}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}/\sqrt{2}\), for which \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{q}}\vphantom{q}}{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{q}}\vphantom{q}}}^{\top} = \mathbf{P}\), and apply projecting with an orthonormal basis:

\[ \mathbf{P} = \frac{1}{2} \begin{bmatrix} 1 & 1 & 0 \\ 1 & 1 & 0 \\ 0 & 0 & 0 \end{bmatrix}. \]

Its diagonal is \((\frac{1}{2}, \frac{1}{2}, 0)\). Each entry is between \(0\) and \(1\), as Theorem 6 says. The entry \(0\) belongs to a coordinate that \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}\) does not involve. For \(\mathbf{P} = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}\) from Example 12 the diagonal is \((1, 0)\), so \(p_{ii}\) can be \(0\) and can be \(1\).

DefDefinition 10 (Orthogonal matrix)

Definition 10 (Orthogonal matrix) A \(p \times p\) matrix \(\mathbf{Q}\) is orthogonal [5] if

\[ \underbrace{{\mathbf{Q}}^{\top}}_{p \times p}\,\underbrace{\mathbf{Q}}_{p \times p} = \mathbf{I}_p \]

ExmExample 14 (A rotation matrix)

Example 14 (A rotation matrix) The matrix

\[ \mathbf{Q} = \begin{bmatrix} 0.6 & -0.8 \\ 0.8 & 0.6 \end{bmatrix} \]

rotates each vector in the plane counterclockwise by the angle \(\theta\) with \(\cos\theta= 0.6\) and \(\sin\theta= 0.8\) (about \(53\) degrees) [6, Ch. 8], Example 8.1, p. 207. It is orthogonal:

\[ \begin{aligned} {\mathbf{Q}}^{\top}\mathbf{Q} &= \begin{bmatrix} 0.6 & 0.8 \\ -0.8 & 0.6 \end{bmatrix} \begin{bmatrix} 0.6 & -0.8 \\ 0.8 & 0.6 \end{bmatrix} && \text{(definition of the transpose)} \\ &= \begin{bmatrix} 0.36 + 0.64 & -0.48 + 0.48 \\ -0.48 + 0.48 & 0.64 + 0.36 \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} && \text{(add)} \end{aligned} \]

ExmExample 15 (Orthogonal columns are not enough)

Example 15 (Orthogonal columns are not enough) \(\mathbf{Q} = 2 \mathbf{I}_2\) has orthogonal columns \((2, 0)\) and \((0, 2)\), but

\[ \begin{aligned} {\mathbf{Q}}^{\top} \mathbf{Q} &= \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix} \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix} \\ &= \begin{bmatrix} 4 & 0 \\ 0 & 4 \end{bmatrix} \\ &\ne \mathbf{I}_2, \end{aligned} \]

so \(\mathbf{Q}\) is not an orthogonal matrix: its columns have norm \(2\), not \(1\).

RemarkRemark 6 (The columns of an orthogonal matrix are orthonormal)

Remark 6 (The columns of an orthogonal matrix are orthonormal). Entry \((i, j)\) of \({\mathbf{Q}}^{\top}\mathbf{Q}\) is the dot product of column \(i\) and column \(j\) of \(\mathbf{Q}\), so \({\mathbf{Q}}^{\top}\mathbf{Q} = \mathbf{I}_p\) says that the columns of \(\mathbf{Q}\) are orthonormal (Definition 28 in Vectors). For a square matrix, \({\mathbf{Q}}^{\top}\mathbf{Q} = \mathbf{I}_p\) also implies \(\mathbf{Q}{\mathbf{Q}}^{\top} = \mathbf{I}_p\), so \(\mathbf{Q}^{-1} = {\mathbf{Q}}^{\top}\) (Definition 5) [6, Ch. 8], Theorem 8.1 and Definition 8.1, p. 209.

For example, the columns of \(\mathbf{Q}\) in Example 14 are \((0.6, 0.8)\) and \((-0.8, 0.6)\). The diagonal entries \(0.36 + 0.64 = 1\) of \({\mathbf{Q}}^{\top}\mathbf{Q}\) are their squared norms, and the off-diagonal entries \(-0.48 + 0.48 = 0\) are their dot product. Multiplying in the other order also gives \(\mathbf{I}_2\):

\[ \begin{aligned} \mathbf{Q}{\mathbf{Q}}^{\top} &= \begin{bmatrix} 0.6 & -0.8 \\ 0.8 & 0.6 \end{bmatrix} \begin{bmatrix} 0.6 & 0.8 \\ -0.8 & 0.6 \end{bmatrix} && \text{(definition of the transpose)} \\ &= \begin{bmatrix} 0.36 + 0.64 & 0.48 - 0.48 \\ 0.48 - 0.48 & 0.64 + 0.36 \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} && \text{(add)} \end{aligned} \]

So \(\mathbf{Q}^{-1} = {\mathbf{Q}}^{\top}\).

LemTheorem 7 (Orthogonal matrices preserve length)

Theorem 7 (Orthogonal matrices preserve length) If \(\mathbf{Q}\) is a \(p \times p\) orthogonal matrix (Definition 10) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) is a vector of length \(p\), then

\[ \mathopen{}\left\lVert\mathbf{Q}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{} = \mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{} \]

Proof. \[ \begin{aligned} \mathopen{}\left\lVert\mathbf{Q}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}^2 &= (\mathbf{Q}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) \cdot (\mathbf{Q}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) && \text{(definition of the Euclidean norm)} \\ &= {(\mathbf{Q}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})}^{\top}\,(\mathbf{Q}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) && \text{(dot product as a matrix product)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\,{\mathbf{Q}}^{\top}\,(\mathbf{Q}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) && \text{(transpose of a product)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\,({\mathbf{Q}}^{\top}\mathbf{Q})\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(regroup; matrix multiplication is associative)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\,\mathbf{I}_p\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(definition of an orthogonal matrix)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(identity matrix)} \\ &= \mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}^2 && \text{(definition of the Euclidean norm)} \end{aligned} \]

Both norms are nonnegative square roots, so equal squares give \(\mathopen{}\left\lVert\mathbf{Q}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{} = \mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}\). The second step is Example 3 in Matrix Multiplication, the third is Theorem 6 in Matrix Multiplication, and the fourth is Theorem 1 in Matrix Multiplication.

ExmExample 16 (Rotating a vector keeps its length)

Example 16 (Rotating a vector keeps its length) With \(\mathbf{Q}\) from Example 14 and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (3, 4)\) from Example 18 in Vectors:

\[ \begin{aligned} \mathbf{Q}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= \begin{bmatrix} 0.6 \cdot 3 - 0.8 \cdot 4 \\ 0.8 \cdot 3 + 0.6 \cdot 4 \end{bmatrix} && \text{(definition of matrix-vector multiplication)} \\ &= \begin{bmatrix} 1.8 - 3.2 \\ 2.4 + 2.4 \end{bmatrix} && \text{(multiply)} \\ &= \begin{bmatrix} -1.4 \\ 4.8 \end{bmatrix} && \text{(add)} \end{aligned} \]

\[ \begin{aligned} \mathopen{}\left\lVert\mathbf{Q}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{} &= \sqrt{(-1.4)^2 + 4.8^2} && \text{(definition of the norm)} \\ &= \sqrt{1.96 + 23.04} && \text{(square)} \\ &= \sqrt{25} && \text{(add)} \\ &= 5 && \text{(take the square root)} \end{aligned} \]

which is \(\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{} = 5\).

2 Quadratic Forms

DefDefinition 11 (Bilinear form)

Definition 11 (Bilinear form) The bilinear form of a \(p \times p\) matrix \(\mathbf{A}\) [7] is the function \({B_{\mathbf{A}}}\) that maps each pair of vectors \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^p\) to the scalar

\[ {{B_{\mathbf{A}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}. \]

ExmExample 17 (A bilinear form that depends on the order of its vectors)

Example 17 (A bilinear form that depends on the order of its vectors)  

Let

\[ \mathbf{A} = \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix}, \quad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \begin{bmatrix} 1 \\ 2 \end{bmatrix}, \quad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= \begin{bmatrix} 3 \\ -1 \end{bmatrix}. \]

The bilinear form of \(\mathbf{A}\) at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\) is

\[ \begin{aligned} {{B_{\mathbf{A}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right)\mathclose{}} &= \begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix} \begin{bmatrix} 3 \\ -1 \end{bmatrix} && \text{(}\href{#def-bilinear-form}{\text{Definition~11}}\text{)} \\ &= \begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 1 \cdot 3 + 2 \cdot(-1) \\ 0 \cdot 3 + 3 \cdot(-1) \end{bmatrix} && \text{(multiply the matrix by } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\text{)} \\ &= \begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 1 \\ -3 \end{bmatrix} && \text{(multiply and add)} \\ &= 1 \cdot 1 + 2 \cdot(-3) && \text{(multiply the row by the column)} \\ &= -5. && \text{(multiply and add)} \end{aligned} \]

With the vectors swapped, it is

\[ \begin{aligned} {{B_{\mathbf{A}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{}} &= \begin{bmatrix} 3 & -1 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} && \text{(}\href{#def-bilinear-form}{\text{Definition~11}}\text{)} \\ &= \begin{bmatrix} 3 & -1 \end{bmatrix} \begin{bmatrix} 1 \cdot 1 + 2 \cdot 2 \\ 0 \cdot 1 + 3 \cdot 2 \end{bmatrix} && \text{(multiply the matrix by } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\text{)} \\ &= \begin{bmatrix} 3 & -1 \end{bmatrix} \begin{bmatrix} 5 \\ 6 \end{bmatrix} && \text{(multiply and add)} \\ &= 3 \cdot 5 + (-1) \cdot 6 && \text{(multiply the row by the column)} \\ &= 9. && \text{(multiply and add)} \end{aligned} \]

The two orders give different values.

DefDefinition 12 (Symmetric bilinear form)

Definition 12 (Symmetric bilinear form) A symmetric bilinear form [7] is a bilinear form \({B_{\mathbf{A}}}\) whose matrix \(\mathbf{A}\) is symmetric:

\[ {\mathbf{A}}^{\top} = \mathbf{A}. \]

LemTheorem 8 (A symmetric bilinear form does not depend on the order of its vectors)

Theorem 8 (A symmetric bilinear form does not depend on the order of its vectors) If \(\mathbf{A}\) is a \(p \times p\) symmetric matrix, then its bilinear form is the same with its vectors swapped, for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^p\):

\[ {{B_{\mathbf{A}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{}} = {{B_{\mathbf{A}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right)\mathclose{}}. \]

Proof. Swapping the vectors relies on the fact that a \(1 \times 1\) product equals its own transpose, which justifies the step labeled with that fact:

\[ \begin{aligned} {{B_{\mathbf{A}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{}} &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}}^{\top}\, \mathbf{A}\, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(}\href{#def-bilinear-form}{\text{Definition~11}}\text{)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\, {\mathbf{A}}^{\top}\, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} && \text{(a } 1 \times 1 \text{ product equals its own transpose)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\, \mathbf{A}\, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} && \text{(}\href{#def-symmetric-matrix}{\text{Definition~3}}\text{)} \\ &= {{B_{\mathbf{A}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right)\mathclose{}}. && \text{(}\href{#def-bilinear-form}{\text{Definition~11}}\text{)} \end{aligned} \]

ExmExample 18 (A symmetric bilinear form)

Example 18 (A symmetric bilinear form)  

Let

\[ \mathbf{S} = \begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix}, \quad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \begin{bmatrix} 1 \\ 2 \end{bmatrix}, \quad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= \begin{bmatrix} 3 \\ -1 \end{bmatrix}. \]

The matrix \(\mathbf{S}\) is symmetric, because its two off-diagonal entries are both \(2\), so its bilinear form is a symmetric bilinear form. At \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\), that form is

\[ \begin{aligned} {{B_{\mathbf{S}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right)\mathclose{}} &= \begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix} \begin{bmatrix} 3 \\ -1 \end{bmatrix} && \text{(}\href{#def-bilinear-form}{\text{Definition~11}}\text{)} \\ &= \begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 1 \cdot 3 + 2 \cdot(-1) \\ 2 \cdot 3 + 1 \cdot(-1) \end{bmatrix} && \text{(multiply the matrix by } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\text{)} \\ &= \begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 1 \\ 5 \end{bmatrix} && \text{(multiply and add)} \\ &= 1 \cdot 1 + 2 \cdot 5 && \text{(multiply the row by the column)} \\ &= 11. && \text{(multiply and add)} \end{aligned} \]

With the vectors swapped, it is

\[ \begin{aligned} {{B_{\mathbf{S}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{}} &= \begin{bmatrix} 3 & -1 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} && \text{(}\href{#def-bilinear-form}{\text{Definition~11}}\text{)} \\ &= \begin{bmatrix} 3 & -1 \end{bmatrix} \begin{bmatrix} 1 \cdot 1 + 2 \cdot 2 \\ 2 \cdot 1 + 1 \cdot 2 \end{bmatrix} && \text{(multiply the matrix by } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\text{)} \\ &= \begin{bmatrix} 3 & -1 \end{bmatrix} \begin{bmatrix} 5 \\ 4 \end{bmatrix} && \text{(multiply and add)} \\ &= 3 \cdot 5 + (-1) \cdot 4 && \text{(multiply the row by the column)} \\ &= 11, && \text{(multiply and add)} \end{aligned} \]

the same value, as Theorem 8 says it must be. By contrast, \(\mathbf{A}\) in Example 17 is not symmetric, so its bilinear form is not a symmetric bilinear form, and swapping the vectors changed its value from \(-5\) to \(9\).

DefDefinition 13 (Quadratic form)

Definition 13 (Quadratic form) A quadratic form [8] is a mathematical expression of the structure

\[{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\, \mathbf{S}\, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\]

where \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) is a \(p \times 1\) vector and \(\mathbf{S}\) is a \(p \times p\) matrix.

RemarkRemark 7 (Quadratic forms in statistics)

Remark 7 (Quadratic forms in statistics). A quadratic form extends the scalar expression \(c x^2\) to vectors: with \(p = 1\) and \(\mathbf{S} = [c]\), \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\,\mathbf{S}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= c x^2\). With \(p = 2\),

\[ {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} = x_1^2 + 2 x_1 x_2 + 3 x_2^2, \]

which is \(1 + 4 + 12 = 17\) at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 2)\).

Quadratic forms occur often in statistics:

  • The residual sum of squares in linear regression (see Vector Calculus) is a quadratic form.
  • The variance of a linear combination of estimates (see Inference about Gaussian Linear Regression Models) is a quadratic form: \({\operatorname{Var}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} \cdot \hat{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}}\right)\mathclose{}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\,{\operatorname{Var}\mathopen{}\left(\hat{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}}\right)\mathclose{}}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\).
RemarkRemark 8 (A quadratic form is a bilinear form with equal vectors)

Remark 8 (A quadratic form is a bilinear form with equal vectors).

Setting \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) in the bilinear form of \(\mathbf{S}\) gives the quadratic form of \(\mathbf{S}\):

\[ {{B_{\mathbf{S}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{S} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}. \]

LemTheorem 9 (A quadratic form is additive in its matrix)

Theorem 9 (A quadratic form is additive in its matrix) If \(\mathbf{A}\) and \(\mathbf{B}\) are \(p \times p\) matrices, \(c\) is a number and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) is a vector of length \(p\), then

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}(\mathbf{A} + \mathbf{B})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}&= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{B} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \\ {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}(c\,\mathbf{A})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}&= c\,{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}. \end{aligned} \]

So for a fixed \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\), the map \(\mathbf{S} \mapsto {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{S} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) is linear in \(\mathbf{S}\).

Proof. By the distributive law (Theorem 2 in Matrix Multiplication), applied on the left and then on the right,

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}(\mathbf{A} + \mathbf{B})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= \mathopen{}\left({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\mathbf{A} + {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\mathbf{B}\right)\mathclose{}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(distribute } {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \text{ over the sum)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{B} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(distribute } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\text{ over the sum)} \end{aligned} \]

For the scalar, a number can be moved out of any product, so \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}(c\,\mathbf{A})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= c\,{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\).

ExmExample 19 (Combining two quadratic forms into one)

Example 19 (Combining two quadratic forms into one) Take \(\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix}\), \(\mathbf{B} = \mathbf{I}\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 2)\). Then

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= 1 + 4 + 12 = 17, \\ {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{I} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= 1 + 4 = 5, \\ {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}(\mathbf{A} + \mathbf{I})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\begin{bmatrix} 2 & 2 \\ 0 & 4 \end{bmatrix}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\\ &= 2 + 4 + 16 \\ &= 22, \end{aligned} \]

which equals \(17 + 5\). In particular, \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}(\mathbf{A} + c\,\mathbf{I})\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + c\,\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}^2\) for every number \(c\), because \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{I} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}^2\). This is how two quadratic terms in the same vector combine into a single quadratic form.

DefDefinition 14 (Symmetric part of a square matrix)

Definition 14 (Symmetric part of a square matrix) The symmetric part of a \(p \times p\) matrix \(\mathbf{S}\) is

\[\frac{1}{2}\mathopen{}\left(\mathbf{S} + {\mathbf{S}}^{\top}\right)\mathclose{}\]

ExmExample 20 (The symmetric part of a \(2 \times 2\) matrix)

Example 20 (The symmetric part of a \(2 \times 2\) matrix) For \(\mathbf{S} = \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix}\), the symmetric part is

\[ \begin{aligned} \frac{1}{2}\mathopen{}\left( \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix} + \begin{bmatrix} 1 & 0 \\ 2 & 3 \end{bmatrix} \right)\mathclose{} &= \frac{1}{2}\begin{bmatrix} 2 & 2 \\ 2 & 6 \end{bmatrix} \\ &= \begin{bmatrix} 1 & 1 \\ 1 & 3 \end{bmatrix}, \end{aligned} \]

which is symmetric.

LemTheorem 10 (A quadratic form depends only on the symmetric part)

Theorem 10 (A quadratic form depends only on the symmetric part) If \(\mathbf{S}\) is a \(p \times p\) matrix and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) is a vector of length \(p\), then

\[ {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{S} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\mathopen{}\left(\frac{1}{2}(\mathbf{S}+{\mathbf{S}}^{\top})\right)\mathclose{}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}. \]

So the value of a quadratic form depends only on the symmetric part (Definition 14) of \(\mathbf{S}\).

Proof. The quadratic form \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{S} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) is a \(1 \times 1\) matrix, so it equals its own transpose:

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{S} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= {\mathopen{}\left({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{S} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{}}^{\top} && \text{(a } 1 \times 1 \text{ matrix is symmetric)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\,{\mathbf{S}}^{\top}\,{\mathopen{}\left({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\right)\mathclose{}}^{\top} && \text{(transpose of a product, twice)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\,{\mathbf{S}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(transposing twice changes nothing)} \end{aligned} \]

Averaging the two expressions for \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{S} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\):

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{S} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= \frac{1}{2}\mathopen{}\left({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{S} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\,{\mathbf{S}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{} && \text{(average of two equal numbers)} \\ &= \frac{1}{2}\,{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\mathopen{}\left(\mathbf{S} + {\mathbf{S}}^{\top}\right)\mathclose{}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(distributive law)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\mathopen{}\left(\frac{1}{2}(\mathbf{S}+{\mathbf{S}}^{\top})\right)\mathclose{}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(move the scalar } \tfrac{1}{2} \text{ inside)} \end{aligned} \]

The distributive step is Theorem 2 in Matrix Multiplication, and the transpose step is Theorem 6 in Matrix Multiplication.

ExmExample 21 (Replacing a matrix by its symmetric part)

Example 21 (Replacing a matrix by its symmetric part) With \(\mathbf{S} = \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix}\) from Example 20 and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 1)\):

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{S} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= 1 + 2 + 0 + 3 \\ &= 6 \\ {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\begin{bmatrix} 1 & 1 \\ 1 & 3 \end{bmatrix}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= 1 + 1 + 1 + 3 \\ &= 6 \end{aligned} \]

Both quadratic forms equal the sum of their matrix’s entries, because every entry of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) is \(1\).

3 Trace and Matrix Inner Product

DefDefinition 15 (Trace)

Definition 15 (Trace) The trace [9] of a \(p \times p\) matrix \(\mathbf{M}\) is the sum of its diagonal entries:

\[\operatorname{tr}(\mathbf{M}) \stackrel{\text{def}}{=}\sum_{i=1}^pM_{ii}\]

ExmExample 22 (The trace of a \(2 \times 2\) matrix)

Example 22 (The trace of a \(2 \times 2\) matrix) \[ \begin{aligned} {\operatorname{tr}\mathopen{}\left(\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}\right)\mathclose{}} &= 1 + 4 \\ &= 5 \end{aligned} \]

LemTheorem 11 (The trace of a product does not depend on the order)

Theorem 11 (The trace of a product does not depend on the order) For an \(m \times n\) matrix \(\mathbf{A}\) and an \(n \times m\) matrix \(\mathbf{B}\):

\[ {\operatorname{tr}\mathopen{}\left(\underbrace{\mathbf{A}\mathbf{B}}_{m \times m}\right)\mathclose{}} = {\operatorname{tr}\mathopen{}\left(\underbrace{\mathbf{B}\mathbf{A}}_{n \times n}\right)\mathclose{}} \]

Proof. \[ \begin{aligned} \operatorname{tr}(\mathbf{A}\mathbf{B}) &= \sum_{i=1}^{m} (\mathbf{A}\mathbf{B})_{ii} && \text{(definition of the trace)} \\ &= \sum_{i=1}^{m} \sum_{s=1}^{n} a_{is}\, b_{si} && \text{(definition of matrix multiplication)} \\ &= \sum_{s=1}^{n} \sum_{i=1}^{m} a_{is}\, b_{si} && \text{(swap the order of two finite sums)} \\ &= \sum_{s=1}^{n} \sum_{i=1}^{m} b_{si}\, a_{is} && \text{(multiplication of numbers is commutative)} \\ &= \sum_{s=1}^{n} (\mathbf{B}\mathbf{A})_{ss} && \text{(definition of matrix multiplication)} \\ &= \operatorname{tr}(\mathbf{B}\mathbf{A}) && \text{(definition of the trace)} \end{aligned} \]

ExmExample 23 (Traces of the two products of a \(2 \times 3\) and a \(3 \times 2\) matrix)

Example 23 (Traces of the two products of a \(2 \times 3\) and a \(3 \times 2\) matrix) Let

\[ \mathbf{A} = \begin{bmatrix} 1 & 2 & 0 \\ 0 & 1 & 3 \end{bmatrix} \qquad \mathbf{B} = \begin{bmatrix} 1 & 0 \\ 2 & 1 \\ 0 & 1 \end{bmatrix} \]

The products are

\[ \mathbf{A}\mathbf{B} = \begin{bmatrix} 5 & 2 \\ 2 & 4 \end{bmatrix} \qquad \mathbf{B}\mathbf{A} = \begin{bmatrix} 1 & 2 & 0 \\ 2 & 5 & 3 \\ 0 & 1 & 3 \end{bmatrix} \]

The trace needs only the diagonal entries. For \(\mathbf{A}\mathbf{B}\), which is \(2 \times 2\):

\[ \begin{aligned} \operatorname{tr}(\mathbf{A}\mathbf{B}) &= (\mathbf{A}\mathbf{B})_{11} + (\mathbf{A}\mathbf{B})_{22} && \text{(definition of the trace)} \\ &= (1 \cdot 1 + 2 \cdot 2 + 0 \cdot 0) + (0 \cdot 0 + 1 \cdot 1 + 3 \cdot 1) && \text{(definition of matrix multiplication)} \\ &= (1 + 4 + 0) + (0 + 1 + 3) && \text{(multiply)} \\ &= 5 + 4 && \text{(add within each entry)} \\ &= 9 && \text{(add)} \end{aligned} \]

For \(\mathbf{B}\mathbf{A}\), which is \(3 \times 3\):

\[ \begin{aligned} \operatorname{tr}(\mathbf{B}\mathbf{A}) &= (\mathbf{B}\mathbf{A})_{11} + (\mathbf{B}\mathbf{A})_{22} + (\mathbf{B}\mathbf{A})_{33} && \text{(definition of the trace)} \\ &= (1 \cdot 1 + 0 \cdot 0) + (2 \cdot 2 + 1 \cdot 1) + (0 \cdot 0 + 1 \cdot 3) && \text{(definition of matrix multiplication)} \\ &= (1 + 0) + (4 + 1) + (0 + 3) && \text{(multiply)} \\ &= 1 + 5 + 3 && \text{(add within each entry)} \\ &= 9 && \text{(add)} \end{aligned} \]

The products have different sizes, but their traces agree.

RemarkRemark 9 (Moving a factor of a triple product)

Remark 9 (Moving a factor of a triple product). Let \(\mathbf{A}\) be \(m \times n\), \(\mathbf{B}\) be \(n \times k\), and \(\mathbf{C}\) be \(k \times m\), so that \(\mathbf{A}\mathbf{B}\) is \(m \times k\) and \(\mathbf{A}\mathbf{B}\mathbf{C}\) is square. Applying Theorem 11 to the \(m \times k\) matrix \(\mathbf{A}\mathbf{B}\) and the \(k \times m\) matrix \(\mathbf{C}\) moves the last factor of the triple product to the front [6, Ch. 1], Theorem 1.5 and eq. 1.17, p. 19:

\[ \begin{aligned} \operatorname{tr}(\mathbf{A}\mathbf{B}\mathbf{C}) &= {\operatorname{tr}\mathopen{}\left((\mathbf{A}\mathbf{B})\,\mathbf{C}\right)\mathclose{}} && \text{(matrix multiplication is associative)} \\ &= {\operatorname{tr}\mathopen{}\left(\mathbf{C}\,(\mathbf{A}\mathbf{B})\right)\mathclose{}} && \text{(the theorem, applied to the two factors } \mathbf{A}\mathbf{B} \text{ and } \mathbf{C} \text{)} \\ &= \operatorname{tr}(\mathbf{C}\mathbf{A}\mathbf{B}) && \text{(matrix multiplication is associative)} \end{aligned} \]

For example, let

\[ \mathbf{A} = \begin{bmatrix} 1 & 2 \\ 0 & 1 \end{bmatrix} \qquad \mathbf{B} = \begin{bmatrix} 1 & 0 \\ 1 & 1 \end{bmatrix} \qquad \mathbf{C} = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} \]

Then

\[ \mathbf{A}\mathbf{B} = \begin{bmatrix} 1 \cdot 1 + 2 \cdot 1 & 1 \cdot 0 + 2 \cdot 1 \\ 0 \cdot 1 + 1 \cdot 1 & 0 \cdot 0 + 1 \cdot 1 \end{bmatrix} = \begin{bmatrix} 3 & 2 \\ 1 & 1 \end{bmatrix} \qquad \mathbf{C}\mathbf{A} = \begin{bmatrix} 0 \cdot 1 + 1 \cdot 0 & 0 \cdot 2 + 1 \cdot 1 \\ 1 \cdot 1 + 0 \cdot 0 & 1 \cdot 2 + 0 \cdot 1 \end{bmatrix} = \begin{bmatrix} 0 & 1 \\ 1 & 2 \end{bmatrix} \]

so

\[ (\mathbf{A}\mathbf{B})\,\mathbf{C} = \begin{bmatrix} 3 \cdot 0 + 2 \cdot 1 & 3 \cdot 1 + 2 \cdot 0 \\ 1 \cdot 0 + 1 \cdot 1 & 1 \cdot 1 + 1 \cdot 0 \end{bmatrix} = \begin{bmatrix} 2 & 3 \\ 1 & 1 \end{bmatrix} \qquad (\mathbf{C}\mathbf{A})\,\mathbf{B} = \begin{bmatrix} 0 \cdot 1 + 1 \cdot 1 & 0 \cdot 0 + 1 \cdot 1 \\ 1 \cdot 1 + 2 \cdot 1 & 1 \cdot 0 + 2 \cdot 1 \end{bmatrix} = \begin{bmatrix} 1 & 1 \\ 3 & 2 \end{bmatrix} \]

and both traces are the same:

\[ \begin{aligned} \operatorname{tr}(\mathbf{A}\mathbf{B}\mathbf{C}) &= 2 + 1 \\ &= 3 \end{aligned} \]

and

\[ \begin{aligned} \operatorname{tr}(\mathbf{C}\mathbf{A}\mathbf{B}) &= 1 + 2 \\ &= 3. \end{aligned} \]

LemTheorem 12 (The trace of an outer product is a dot product)

Theorem 12 (The trace of an outer product is a dot product) For two vectors \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) with the same number \(n\) of entries, the trace of the outer product \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\) and the dot product \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) are the same number:

\[ {\operatorname{tr}\mathopen{}\left(\underbrace{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}}_{n \times n}\right)\mathclose{}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \]

Proof. \[ \begin{aligned} {\operatorname{tr}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\right)\mathclose{}} &= \sum_{i=1}^{n} \mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}^{\top}\right)\mathclose{}_{ii} && \text{(definition of the trace)} \\ &= \sum_{i=1}^{n} u_i v_i && \text{(entries of an outer product)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} && \text{(definition of the dot product)} \end{aligned} \]

ExmExample 24 (The trace of an outer product)

Example 24 (The trace of an outer product) For \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}} = (1, 2)\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = (3, 4)\), the remark on outer and dot products found the outer product \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}^{\top}\) and the dot product \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = 11\). The trace of the outer product is

\[ \begin{aligned} {\operatorname{tr}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}^{\top}\right)\mathclose{}} &= {\operatorname{tr}\mathopen{}\left(\begin{bmatrix} 3 & 4 \\ 6 & 8 \end{bmatrix}\right)\mathclose{}} && \text{(the outer product found in the remark)} \\ &= 3 + 8 && \text{(definition of the trace)} \\ &= 11 && \text{(add)} \end{aligned} \]

The trace and the dot product are the same number, as Theorem 12 says.

CorCorollary 3 (The trace of a self outer product is a squared norm)

Corollary 3 (The trace of a self outer product is a squared norm) For a vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\), the trace of the self outer product \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\) and the square of the Euclidean norm \(\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}\) are the same number:

\[ {\operatorname{tr}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\right)\mathclose{}} = \mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}^2 \]

Proof. \[ \begin{aligned} {\operatorname{tr}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\right)\mathclose{}} &= {\operatorname{tr}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\right)\mathclose{}} && \text{(definition of the self outer product)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(the trace of an outer product is a dot product)} \\ &= \mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}^2 && \text{(the Euclidean norm from the dot product, squared)} \end{aligned} \]

The second step is Theorem 12, with \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) in place of both \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\). The third step takes the square of both sides of the formula for the Euclidean norm from the dot product.

ExmExample 25 (The trace of a self outer product)

Example 25 (The trace of a self outer product) For \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 2)\), the example of a self outer product found \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} = \begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix}\). Its trace and the squared Euclidean norm of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) are

\[ \begin{aligned} {\operatorname{tr}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\right)\mathclose{}} &= 1 + 4 && \text{(definition of the trace)} \\ &= 5 && \text{(add)} \\ &= 1^2 + 2^2 && \text{(write each diagonal entry as a square)} \\ &= \mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}^2 && \text{(definition of the Euclidean norm, squared)} \end{aligned} \]

as Corollary 3 says.

DefDefinition 16 (Matrix inner product)

Definition 16 (Matrix inner product) The inner product of two \(n \times p\) matrices \(\mathbf{A}\) and \(\mathbf{B}\) is

\[ {\mathopen{}\left\langle \mathbf{A}, \mathbf{B}\right\rangle\mathclose{}} \stackrel{\text{def}}{=} {\operatorname{tr}\mathopen{}\left(\underbrace{{\mathbf{A}}^{\top}\mathbf{B}}_{p \times p}\right)\mathclose{}} \]

Banerjee and Roy [6, Ch. 15], eq. 15.7, p. 491 defines the same inner product on \(n \times p\) matrices with the trace.

LemTheorem 13 (The matrix inner product multiplies matching entries)

Theorem 13 (The matrix inner product multiplies matching entries) For two \(n \times p\) matrices \(\mathbf{A}\) and \(\mathbf{B}\):

\[ {\mathopen{}\left\langle \mathbf{A}, \mathbf{B}\right\rangle\mathclose{}} = \sum_{i=1}^n\sum_{j=1}^pa_{ij}\, b_{ij} \]

Proof. \[ \begin{aligned} {\mathopen{}\left\langle \mathbf{A}, \mathbf{B}\right\rangle\mathclose{}} &= \operatorname{tr}({\mathbf{A}}^{\top}\mathbf{B}) && \text{(definition of the inner product)} \\ &= \sum_{j=1}^p({\mathbf{A}}^{\top}\mathbf{B})_{jj} && \text{(definition of the trace)} \\ &= \sum_{j=1}^p\sum_{i=1}^n({\mathbf{A}}^{\top})_{ji}\, b_{ij} && \text{(definition of matrix multiplication)} \\ &= \sum_{j=1}^p\sum_{i=1}^na_{ij}\, b_{ij} && \text{(definition of the transpose)} \\ &= \sum_{i=1}^n\sum_{j=1}^pa_{ij}\, b_{ij} && \text{(swap the order of two finite sums)} \end{aligned} \]

ExmExample 26 (The inner product of two \(2 \times 2\) matrices)

Example 26 (The inner product of two \(2 \times 2\) matrices) Let

\[ \mathbf{A} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \qquad \mathbf{B} = \begin{bmatrix} 0 & 1 \\ -1 & 2 \end{bmatrix} \]

From Definition 16:

\[ \begin{aligned} {\mathopen{}\left\langle \mathbf{A}, \mathbf{B}\right\rangle\mathclose{}} &= {\operatorname{tr}\mathopen{}\left({\mathbf{A}}^{\top}\mathbf{B}\right)\mathclose{}} && \text{(definition of the inner product)} \\ &= {\operatorname{tr}\mathopen{}\left( {\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}}^{\top} \begin{bmatrix} 0 & 1 \\ -1 & 2 \end{bmatrix} \right)\mathclose{}} && \text{(substitute)} \\ &= {\operatorname{tr}\mathopen{}\left( \begin{bmatrix} 1 & 3 \\ 2 & 4 \end{bmatrix} \begin{bmatrix} 0 & 1 \\ -1 & 2 \end{bmatrix} \right)\mathclose{}} && \text{(definition of the transpose)} \\ &= {\operatorname{tr}\mathopen{}\left(\begin{bmatrix} -3 & 7 \\ -4 & 10 \end{bmatrix}\right)\mathclose{}} && \text{(multiply)} \\ &= -3 + 10 && \text{(definition of the trace)} \\ &= 7 && \text{(add)} \end{aligned} \]

From Theorem 13:

\[ \begin{aligned} {\mathopen{}\left\langle \mathbf{A}, \mathbf{B}\right\rangle\mathclose{}} &= 1 \cdot 0 + 2 \cdot 1 + 3 \cdot(-1) + 4 \cdot 2 && \text{(multiply matching entries and add)} \\ &= 0 + 2 - 3 + 8 && \text{(multiply)} \\ &= 7 && \text{(add)} \end{aligned} \]

RemarkRemark 10 (The matrix inner product as a dot product)

Remark 10 (The matrix inner product as a dot product). Theorem 13 says that the matrix inner product is the dot product (Definition 10 in Vectors) of the two matrices’ entries, each listed as one vector of length \(np\), with both matrices’ entries listed in the same order.

For example, listing the entries of \(\mathbf{A}\) and \(\mathbf{B}\) in Example 26 row by row gives \((1, 2, 3, 4)\) and \((0, 1, -1, 2)\), and

\[ \begin{aligned} (1, 2, 3, 4) \cdot (0, 1, -1, 2) &= 7 \\ &= {\mathopen{}\left\langle \mathbf{A}, \mathbf{B}\right\rangle\mathclose{}}. \end{aligned} \]

DefDefinition 17 (Frobenius norm)

Definition 17 (Frobenius norm) The Frobenius norm [10] of an \(n \times p\) matrix \(\mathbf{A}\) is

\[ \mathopen{}\left\lVert\mathbf{A}\right\rVert\mathclose{}_F \stackrel{\text{def}}{=}\sqrt{{\mathopen{}\left\langle \mathbf{A}, \mathbf{A}\right\rangle\mathclose{}}} \]

where \({\mathopen{}\left\langle \cdot, \cdot\right\rangle\mathclose{}}\) is the matrix inner product (Definition 16).

ExmExample 27 (The Frobenius norm of a \(2 \times 2\) matrix)

Example 27 (The Frobenius norm of a \(2 \times 2\) matrix) For \(\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}\):

\[ \begin{aligned} \mathopen{}\left\lVert\mathbf{A}\right\rVert\mathclose{}_F &= \sqrt{{\mathopen{}\left\langle \mathbf{A}, \mathbf{A}\right\rangle\mathclose{}}} && \text{(definition of the Frobenius norm)} \\ &= \sqrt{1 \cdot 1 + 2 \cdot 2 + 3 \cdot 3 + 4 \cdot 4} && \text{(sum of products of matching entries)} \\ &= \sqrt{1 + 4 + 9 + 16} && \text{(multiply)} \\ &= \sqrt{30} && \text{(add)} \end{aligned} \]

The second step is Theorem 13.

RemarkRemark 11 (The Frobenius norm is the Euclidean norm of the entries)

Remark 11 (The Frobenius norm is the Euclidean norm of the entries). By Theorem 13, \(\mathopen{}\left\lVert\mathbf{A}\right\rVert\mathclose{}_F^2 = \sum_{i=1}^n\sum_{j=1}^pa_{ij}^2\), a sum of squares, so the square root is always defined. \(\mathopen{}\left\lVert\mathbf{A}\right\rVert\mathclose{}_F\) is the Euclidean norm (Definition 23 in Vectors) of the entries of \(\mathbf{A}\), listed as one vector of length \(np\) [6, Ch. 15], Definition 15.4, p. 492.

For example, listing the entries of \(\mathbf{A}\) in Example 27 row by row gives the vector \((1, 2, 3, 4)\) of length \(2 \cdot 2 = 4\), and

\[ \begin{aligned} \mathopen{}\left\lVert(1, 2, 3, 4)\right\rVert\mathclose{} &= \sqrt{1 + 4 + 9 + 16} \\ &= \sqrt{30} \\ &= \mathopen{}\left\lVert\mathbf{A}\right\rVert\mathclose{}_F. \end{aligned} \]

LemTheorem 14 (The squared Frobenius norm adds up the squared column norms)

Theorem 14 (The squared Frobenius norm adds up the squared column norms) If \(\mathbf{A}\) is an \(n \times p\) matrix with columns \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_1, \ldots, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_p\), then

\[ \mathopen{}\left\lVert\mathbf{A}\right\rVert\mathclose{}_F^2 = \sum_{j=1}^p\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_j\right\rVert\mathclose{}^2. \]

Proof. Entry \(i\) of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_j\) is \(a_{ij}\), so

\[ \begin{aligned} \mathopen{}\left\lVert\mathbf{A}\right\rVert\mathclose{}_F^2 &= \sum_{i=1}^n\sum_{j=1}^pa_{ij}^2 && \text{(}\href{#rem-frobenius-norm-entries}{\text{Remark~11}}\text{)} \\ &= \sum_{j=1}^p\sum_{i=1}^na_{ij}^2 && \text{(swap the order of two finite sums)} \\ &= \sum_{j=1}^p\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_j\right\rVert\mathclose{}^2 && \text{(}\href{linear-algebra-vectors.qmd#def-euclidean-norm}{\text{Definition~23 in Vectors}}\text{, applied to each column)} \end{aligned} \]

ExmExample 28 (The Frobenius norm of a \(2 \times 2\) matrix, column by column)

Example 28 (The Frobenius norm of a \(2 \times 2\) matrix, column by column) The matrix \(\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}\) in Example 27 has columns \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_1 = (1, 3)\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_2 = (2, 4)\), so

\[ \begin{aligned} \mathopen{}\left\lVert\mathbf{A}\right\rVert\mathclose{}_F^2 &= \mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_1\right\rVert\mathclose{}^2 + \mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_2\right\rVert\mathclose{}^2 && \text{(}\href{#thm-frobenius-norm-columns}{\text{Theorem~14}}\text{)} \\ &= (1^2 + 3^2) + (2^2 + 4^2) && \text{(definition of the norm)} \\ &= 10 + 20 && \text{(square and add)} \\ &= 30 && \text{(add)} \end{aligned} \]

which matches \(\mathopen{}\left\lVert\mathbf{A}\right\rVert\mathclose{}_F = \sqrt{30}\) from Example 27.

4 Matrix Decompositions

DefDefinition 18 (Eigenvalue)

Definition 18 (Eigenvalue) Let \(\mathbf{A}\) be a \(p \times p\) matrix. A real number \(\lambda\) is an eigenvalue of \(\mathbf{A}\) if some real vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) of length \(p\) satisfies

\[ \underbrace{\mathbf{A}}_{p \times p}\,\underbrace{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}_{p \times 1} = \lambda\,\underbrace{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}}_{p \times 1} \]

DefDefinition 19 (Eigenvector)

Definition 19 (Eigenvector) In Definition 18, any such \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) is an eigenvector of \(\mathbf{A}\) for \(\lambda\).

ExmExample 29 (Eigenvectors of a \(2 \times 2\) matrix)

Example 29 (Eigenvectors of a \(2 \times 2\) matrix) Let \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\).

\((1, 1)\) is an eigenvector for the eigenvalue \(3\):

\[ \begin{aligned} \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \begin{bmatrix} 1 \\ 1 \end{bmatrix} &= \begin{bmatrix} 2 + 1 \\ 1 + 2 \end{bmatrix} && \text{(definition of matrix-vector multiplication)} \\ &= 3 \begin{bmatrix} 1 \\ 1 \end{bmatrix} && \text{(factor out } 3 \text{)} \end{aligned} \]

\((1, -1)\) is an eigenvector for the eigenvalue \(1\):

\[ \begin{aligned} \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \begin{bmatrix} 1 \\ -1 \end{bmatrix} &= \begin{bmatrix} 2 - 1 \\ 1 - 2 \end{bmatrix} && \text{(definition of matrix-vector multiplication)} \\ &= 1 \begin{bmatrix} 1 \\ -1 \end{bmatrix} && \text{(factor out } 1 \text{)} \end{aligned} \]

RemarkRemark 12 (Multiples of an eigenvector)

Remark 12 (Multiples of an eigenvector). Multiplying by \(\mathbf{A}\) rescales an eigenvector by \(\lambda\) and does not change the line it lies on. Any nonzero multiple \(c\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) of an eigenvector is also an eigenvector for the same \(\lambda\):

\[ \begin{aligned} \mathbf{A}(c\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}) &= c\,\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} && \text{(move the scalar } c \text{ to the front)} \\ &= c\,\lambda\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} && \text{(} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \text{ is an eigenvector for } \lambda\text{)} \\ &= \lambda\,(c\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}) && \text{(multiplication of numbers is commutative)} \end{aligned} \]

For example, in Example 29, \(2\,(1, 1) = (2, 2)\) is also an eigenvector for the eigenvalue \(3\):

\[ \begin{aligned} \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \begin{bmatrix} 2 \\ 2 \end{bmatrix} &= \begin{bmatrix} 2 \cdot 2 + 1 \cdot 2 \\ 1 \cdot 2 + 2 \cdot 2 \end{bmatrix} && \text{(definition of matrix-vector multiplication)} \\ &= \begin{bmatrix} 4 + 2 \\ 2 + 4 \end{bmatrix} && \text{(multiply)} \\ &= \begin{bmatrix} 6 \\ 6 \end{bmatrix} && \text{(add)} \\ &= 3 \begin{bmatrix} 2 \\ 2 \end{bmatrix} && \text{(factor out } 3 \text{)} \end{aligned} \]

ExmExample 30 (A vector that is not an eigenvector)

Example 30 (A vector that is not an eigenvector) For the same \(\mathbf{A}\), \((1, 0)\) is not an eigenvector:

\[ \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 2 \\ 1 \end{bmatrix}, \]

and no number \(\lambda\) gives \((2, 1) = \lambda\,(1, 0)\), because the second entries would need \(1 = \lambda\cdot 0\).

RemarkRemark 13 (Real and complex eigenvalues)

Remark 13 (Real and complex eigenvalues). These notes take \(\lambda\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) to be real. Banerjee and Roy [6, Ch. 11], Definition 11.1, pp. 312-313 also allows complex eigenvalues and eigenvectors, because some real matrices, such as a rotation by \(90\) degrees, have no real eigenvalues [6, Ch. 11], Example 11.2, p. 312.

For example, \(\mathbf{R} = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}\) rotates each vector in the plane counterclockwise by \(90\) degrees: \(\mathbf{R}\,(v_1, v_2) = (-v_2, v_1)\). Suppose \(\mathbf{R}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = \lambda\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) for a real number \(\lambda\). Matching entries gives \(-v_2 = \lambda v_1\) and \(v_1 = \lambda v_2\). Substituting the second equation into the first gives \(-v_2 = \lambda^2 v_2\), so \((1 + \lambda^2)\, v_2 = 0\). Since \(1 + \lambda^2 > 0\), \(v_2 = 0\), and then

\[ \begin{aligned} v_1 &= \lambda v_2 \\ &= 0. \end{aligned} \]

So the only solution is \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), and \(\mathbf{R}\) has no real eigenvalue.

TipVideo

Chapter 14 of the Essence of linear algebra series by 3Blue1Brown shows the geometric meaning of eigenvectors and eigenvalues:

LemTheorem 15 (Spectral theorem for symmetric matrices)

Theorem 15 (Spectral theorem for symmetric matrices) If \(\mathbf{A}\) is a \(p \times p\) symmetric matrix (Definition 3) with real entries, then there are a \(p \times p\) orthogonal matrix \(\mathbf{Q}\) (Definition 10) and a \(p \times p\) diagonal matrix \(\mathbf{\Lambda}\) (Definition 4) with real diagonal entries \(\lambda_1, \ldots, \lambda_p\) such that

\[ \underbrace{\mathbf{A}}_{p \times p} = \underbrace{\mathbf{Q}}_{p \times p}\, \underbrace{\mathbf{\Lambda}}_{p \times p}\, \underbrace{{\mathbf{Q}}^{\top}}_{p \times p} \]

Each \(\lambda_i\) is an eigenvalue of \(\mathbf{A}\) (Definition 18), and column \(i\) of \(\mathbf{Q}\) is an eigenvector of \(\mathbf{A}\) for \(\lambda_i\).

RemarkRemark 14 (An equivalent form of the spectral theorem)

Remark 14 (An equivalent form of the spectral theorem). The proof, by induction on \(p\), is outside the scope of these notes; see Banerjee and Roy [6, Ch. 11], Theorem 11.27, p. 349, which states the result in the equivalent form \({\mathbf{Q}}^{\top}\mathbf{A}\mathbf{Q} = \mathbf{\Lambda}\). The two forms are equivalent because

\[ \begin{aligned} {\mathbf{Q}}^{\top}\mathbf{Q} &= \mathbf{Q}{\mathbf{Q}}^{\top} \\ &= \mathbf{I}_p \end{aligned} \]

(Remark 6). Multiplying \(\mathbf{A} = \mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top}\) by \({\mathbf{Q}}^{\top}\) on the left and by \(\mathbf{Q}\) on the right gives

\[ \begin{aligned} {\mathbf{Q}}^{\top}\mathbf{A}\mathbf{Q} &= {\mathbf{Q}}^{\top}\mathopen{}\left(\mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top}\right)\mathclose{}\mathbf{Q} && \text{(substitute)} \\ &= \mathopen{}\left({\mathbf{Q}}^{\top}\mathbf{Q}\right)\mathclose{}\mathbf{\Lambda}\mathopen{}\left({\mathbf{Q}}^{\top}\mathbf{Q}\right)\mathclose{} && \text{(regroup; matrix multiplication is associative)} \\ &= \mathbf{I}_p\mathbf{\Lambda}\mathbf{I}_p && \text{(} \mathbf{Q} \text{ is orthogonal)} \\ &= \mathbf{\Lambda} && \text{(multiplying by } \mathbf{I}_p \text{ changes nothing)} \end{aligned} \]

Multiplying \({\mathbf{Q}}^{\top}\mathbf{A}\mathbf{Q} = \mathbf{\Lambda}\) by \(\mathbf{Q}\) on the left and by \({\mathbf{Q}}^{\top}\) on the right gives back the first form:

\[ \begin{aligned} \mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top} &= \mathbf{Q}\mathopen{}\left({\mathbf{Q}}^{\top}\mathbf{A}\mathbf{Q}\right)\mathclose{}{\mathbf{Q}}^{\top} && \text{(substitute)} \\ &= \mathopen{}\left(\mathbf{Q}{\mathbf{Q}}^{\top}\right)\mathclose{}\mathbf{A}\mathopen{}\left(\mathbf{Q}{\mathbf{Q}}^{\top}\right)\mathclose{} && \text{(regroup; matrix multiplication is associative)} \\ &= \mathbf{I}_p\mathbf{A}\mathbf{I}_p && \text{(} \mathbf{Q}{\mathbf{Q}}^{\top} = \mathbf{I}_p \text{)} \\ &= \mathbf{A} && \text{(multiplying by } \mathbf{I}_p \text{ changes nothing)} \end{aligned} \]

For example, let \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\) and \(\mathbf{Q} = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}\), which is symmetric, so \({\mathbf{Q}}^{\top} = \mathbf{Q}\). Then

\[ \begin{aligned} {\mathbf{Q}}^{\top}\mathbf{A}\mathbf{Q} &= \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(substitute)} \\ &= \frac{1}{\sqrt{2}} \cdot\frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(move the scalars to the front)} \\ &= \frac{1}{2} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(multiply the scalars)} \\ &= \frac{1}{2} \begin{bmatrix} 1 \cdot 2 + 1 \cdot 1 & 1 \cdot 1 + 1 \cdot 2 \\ 1 \cdot 2 + (-1) \cdot 1 & 1 \cdot 1 + (-1) \cdot 2 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(definition of matrix multiplication, first two matrices)} \\ &= \frac{1}{2} \begin{bmatrix} 2 + 1 & 1 + 2 \\ 2 - 1 & 1 - 2 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(multiply)} \\ &= \frac{1}{2} \begin{bmatrix} 3 & 3 \\ 1 & -1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(add)} \\ &= \frac{1}{2} \begin{bmatrix} 3 \cdot 1 + 3 \cdot 1 & 3 \cdot 1 + 3 \cdot(-1) \\ 1 \cdot 1 + (-1) \cdot 1 & 1 \cdot 1 + (-1) \cdot(-1) \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \frac{1}{2} \begin{bmatrix} 3 + 3 & 3 - 3 \\ 1 - 1 & 1 + 1 \end{bmatrix} && \text{(multiply)} \\ &= \frac{1}{2} \begin{bmatrix} 6 & 0 \\ 0 & 2 \end{bmatrix} && \text{(add)} \\ &= \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix} && \text{(multiply by } \tfrac{1}{2} \text{)} \end{aligned} \]

a diagonal matrix with the eigenvalues \(3\) and \(1\) of \(\mathbf{A}\) from Example 29 on its diagonal.

LemTheorem 16 (Eigenvectors of a symmetric matrix for distinct eigenvalues are orthogonal)

Theorem 16 (Eigenvectors of a symmetric matrix for distinct eigenvalues are orthogonal) Let \(\mathbf{A}\) be a symmetric \(p \times p\) matrix (Definition 3), and let \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) be eigenvectors of \(\mathbf{A}\) (Definition 19) for eigenvalues \(\lambda_1 \ne \lambda_2\). Then \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} \perp \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) (Definition 22 in Vectors).

Proof. \[ \begin{aligned} \lambda_1\,(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}) &= (\lambda_1 \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}) \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} && \text{(}\href{linear-algebra-vectors.qmd#def-dot-product}{\text{Definition~10 in Vectors}}\text{)} \\ &= (\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}) \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} && \text{(} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} \text{ is an eigenvector for } \lambda_1 \text{)} \\ &= {(\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}})}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} && \text{(}\href{linear-algebra-vectors.qmd#def-dot-product}{\text{Definition~10 in Vectors}}\text{)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\,{\mathbf{A}}^{\top}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} && \text{(}\href{linear-algebra-matrix-multiplication.qmd#thm-transpose-product}{\text{Theorem~6 in Matrix Multiplication}}\text{)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\,\mathbf{A}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} && \text{(} \mathbf{A} \text{ is symmetric)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}}^{\top}\,(\lambda_2 \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}) && \text{(} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \text{ is an eigenvector for } \lambda_2 \text{)} \\ &= \lambda_2\,(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}). && \text{(}\href{linear-algebra-vectors.qmd#def-dot-product}{\text{Definition~10 in Vectors}}\text{)} \end{aligned} \]

So \((\lambda_1 - \lambda_2)\,(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}) = 0\), and \(\lambda_1 - \lambda_2 \ne 0\), so \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = 0\).

ExmExample 31 (Eigenvectors of a symmetric \(2 \times 2\) matrix)

Example 31 (Eigenvectors of a symmetric \(2 \times 2\) matrix) \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\) is symmetric. Its eigenvectors \((1, 1)\) for \(\lambda_1 = 3\) and \((1, -1)\) for \(\lambda_2 = 1\) (Example 29) satisfy

\[ \begin{aligned} (1, 1) \cdot (1, -1) &= 1 \cdot 1 + 1 \cdot(-1) \\ &= 0. \end{aligned} \]

DefDefinition 20 (Eigendecomposition, spectral decomposition)

Definition 20 (Eigendecomposition, spectral decomposition) Let \(\mathbf{A}\) be a \(p \times p\) symmetric matrix with real entries. An eigendecomposition [11], or spectral decomposition, of \(\mathbf{A}\) is a factorization

\[ \mathbf{A} = \mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top} \]

with \(\mathbf{Q}\) a \(p \times p\) orthogonal matrix (Definition 10) and \(\mathbf{\Lambda}\) a \(p \times p\) diagonal matrix (Definition 4).

ExmExample 32 (An eigendecomposition of a \(2 \times 2\) symmetric matrix)

Example 32 (An eigendecomposition of a \(2 \times 2\) symmetric matrix) For \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\), Example 29 found the eigenvectors \((1, 1)\) for \(3\) and \((1, -1)\) for \(1\). They are orthogonal (Definition 22 in Vectors):

\[ \begin{aligned} (1, 1) \cdot (1, -1) &= 1 \cdot 1 + 1 \cdot(-1) && \text{(definition of the dot product)} \\ &= 1 - 1 && \text{(multiply)} \\ &= 0 && \text{(add)} \end{aligned} \]

Each has norm \(\sqrt{1^2 + 1^2} = \sqrt{2}\) (Definition 23 in Vectors), so dividing each by \(\sqrt{2}\) gives two orthonormal eigenvectors (Definition 28 in Vectors). Put them in the columns of \(\mathbf{Q}\), and the matching eigenvalues on the diagonal of \(\mathbf{\Lambda}\):

\[ \mathbf{Q} = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} \qquad \mathbf{\Lambda} = \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix} \]

This \(\mathbf{Q}\) is symmetric, so \({\mathbf{Q}}^{\top} = \mathbf{Q}\). Then

\[ \begin{aligned} \mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top} &= \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix} \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(substitute)} \\ &= \frac{1}{\sqrt{2}} \cdot\frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(move the scalars to the front)} \\ &= \frac{1}{2} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(multiply the scalars)} \\ &= \frac{1}{2} \begin{bmatrix} 1 \cdot 3 + 1 \cdot 0 & 1 \cdot 0 + 1 \cdot 1 \\ 1 \cdot 3 + (-1) \cdot 0 & 1 \cdot 0 + (-1) \cdot 1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(definition of matrix multiplication, first two matrices)} \\ &= \frac{1}{2} \begin{bmatrix} 3 + 0 & 0 + 1 \\ 3 + 0 & 0 - 1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(multiply)} \\ &= \frac{1}{2} \begin{bmatrix} 3 & 1 \\ 3 & -1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(add)} \\ &= \frac{1}{2} \begin{bmatrix} 3 \cdot 1 + 1 \cdot 1 & 3 \cdot 1 + 1 \cdot(-1) \\ 3 \cdot 1 + (-1) \cdot 1 & 3 \cdot 1 + (-1) \cdot(-1) \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \frac{1}{2} \begin{bmatrix} 3 + 1 & 3 - 1 \\ 3 - 1 & 3 + 1 \end{bmatrix} && \text{(multiply)} \\ &= \frac{1}{2} \begin{bmatrix} 4 & 2 \\ 2 & 4 \end{bmatrix} && \text{(add)} \\ &= \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} && \text{(multiply by } \tfrac{1}{2} \text{)} \end{aligned} \]

which is \(\mathbf{A}\).

RemarkRemark 15 (An eigendecomposition is not unique)

Remark 15 (An eigendecomposition is not unique). A symmetric matrix can have more than one eigendecomposition. Reordering the eigenvalues on the diagonal of \(\mathbf{\Lambda}\), and the columns of \(\mathbf{Q}\) with them, gives another one, and so does multiplying a column of \(\mathbf{Q}\) by \(-1\).

For example, take \(\mathbf{A}\) from Example 32. Swapping the two eigenvalues and the two columns gives

\[ \mathbf{Q}_1 = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix} \qquad \mathbf{\Lambda}_1 = \begin{bmatrix} 1 & 0 \\ 0 & 3 \end{bmatrix} \]

and multiplying the second column of \(\mathbf{Q}\) by \(-1\) gives

\[ \begin{aligned} \mathbf{Q}_2 &= \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix} \\ \mathbf{\Lambda}_2 &= \mathbf{\Lambda} \\ &= \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix} \end{aligned} \]

The columns of \(\mathbf{Q}_1\) and \(\mathbf{Q}_2\) are the columns of \(\mathbf{Q}\), reordered or multiplied by \(-1\), so they are still orthonormal, and \(\mathbf{Q}_1\) and \(\mathbf{Q}_2\) are orthogonal. Both products give back \(\mathbf{A}\):

\[ \begin{aligned} \mathbf{Q}_1\mathbf{\Lambda}_1{\mathbf{Q}_1}^{\top} &= \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 3 \end{bmatrix} {\mathopen{}\left(\frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix}\right)\mathclose{}}^{\top} && \text{(substitute)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 3 \end{bmatrix} \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix} && \text{(definition of the transpose)} \\ &= \frac{1}{\sqrt{2}} \cdot\frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 3 \end{bmatrix} \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix} && \text{(move the scalars to the front)} \\ &= \frac{1}{2} \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 3 \end{bmatrix} \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix} && \text{(multiply the scalars)} \\ &= \frac{1}{2} \begin{bmatrix} 1 \cdot 1 + 1 \cdot 0 & 1 \cdot 0 + 1 \cdot 3 \\ -1 \cdot 1 + 1 \cdot 0 & -1 \cdot 0 + 1 \cdot 3 \end{bmatrix} \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix} && \text{(definition of matrix multiplication, first two matrices)} \\ &= \frac{1}{2} \begin{bmatrix} 1 + 0 & 0 + 3 \\ -1 + 0 & 0 + 3 \end{bmatrix} \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix} && \text{(multiply)} \\ &= \frac{1}{2} \begin{bmatrix} 1 & 3 \\ -1 & 3 \end{bmatrix} \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix} && \text{(add)} \\ &= \frac{1}{2} \begin{bmatrix} 1 \cdot 1 + 3 \cdot 1 & 1 \cdot(-1) + 3 \cdot 1 \\ -1 \cdot 1 + 3 \cdot 1 & -1 \cdot(-1) + 3 \cdot 1 \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \frac{1}{2} \begin{bmatrix} 1 + 3 & -1 + 3 \\ -1 + 3 & 1 + 3 \end{bmatrix} && \text{(multiply)} \\ &= \frac{1}{2} \begin{bmatrix} 4 & 2 \\ 2 & 4 \end{bmatrix} && \text{(add)} \\ &= \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} && \text{(multiply by } \tfrac{1}{2} \text{)} \end{aligned} \]

\[ \begin{aligned} \mathbf{Q}_2\mathbf{\Lambda}_2{\mathbf{Q}_2}^{\top} &= \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix} \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix} {\mathopen{}\left(\frac{1}{\sqrt{2}} \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix}\right)\mathclose{}}^{\top} && \text{(substitute)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix} \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix} \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix} && \text{(definition of the transpose)} \\ &= \frac{1}{\sqrt{2}} \cdot\frac{1}{\sqrt{2}} \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix} \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix} && \text{(move the scalars to the front)} \\ &= \frac{1}{2} \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix} \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix} && \text{(multiply the scalars)} \\ &= \frac{1}{2} \begin{bmatrix} 1 \cdot 3 + (-1) \cdot 0 & 1 \cdot 0 + (-1) \cdot 1 \\ 1 \cdot 3 + 1 \cdot 0 & 1 \cdot 0 + 1 \cdot 1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix} && \text{(definition of matrix multiplication, first two matrices)} \\ &= \frac{1}{2} \begin{bmatrix} 3 + 0 & 0 - 1 \\ 3 + 0 & 0 + 1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix} && \text{(multiply)} \\ &= \frac{1}{2} \begin{bmatrix} 3 & -1 \\ 3 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix} && \text{(add)} \\ &= \frac{1}{2} \begin{bmatrix} 3 \cdot 1 + (-1) \cdot(-1) & 3 \cdot 1 + (-1) \cdot 1 \\ 3 \cdot 1 + 1 \cdot(-1) & 3 \cdot 1 + 1 \cdot 1 \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \frac{1}{2} \begin{bmatrix} 3 + 1 & 3 - 1 \\ 3 - 1 & 3 + 1 \end{bmatrix} && \text{(multiply)} \\ &= \frac{1}{2} \begin{bmatrix} 4 & 2 \\ 2 & 4 \end{bmatrix} && \text{(add)} \\ &= \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} && \text{(multiply by } \tfrac{1}{2} \text{)} \end{aligned} \]

LemTheorem 17 (Singular value decomposition)

Theorem 17 (Singular value decomposition) Let \(\mathbf{A}\) be an \(n \times p\) matrix with real entries and \(\operatorname{rank}(\mathbf{A}) = r\) (Definition 2 in Subspaces and Rank). Then there are an \(n \times n\) orthogonal matrix \(\mathbf{U}\), a \(p \times p\) orthogonal matrix \(\mathbf{V}\) (Definition 10), and numbers \(\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_r > 0\) such that

\[ \underbrace{\mathbf{A}}_{n \times p} = \underbrace{\mathbf{U}}_{n \times n}\, \underbrace{\mathbf{D}}_{n \times p}\, \underbrace{{\mathbf{V}}^{\top}}_{p \times p} \]

where \(\mathbf{D}\) has entries \(d_{ii} = \sigma_i\) for \(i = 1, \ldots, r\) and every other entry \(0\).

RemarkRemark 16 (The SVD applies to every real matrix)

Remark 16 (The SVD applies to every real matrix). The proof is outside the scope of these notes; see Banerjee and Roy [6, Ch. 12], Theorem 12.1, p. 373. Unlike the spectral theorem (Theorem 15), Theorem 17 applies to every real matrix, including one that is not square or not symmetric.

For example, the \(1 \times 2\) matrix \(\mathbf{A} = \begin{bmatrix} 1 & 1 \end{bmatrix}\) is not square, so Theorem 15 does not apply to it. It has rank \(1\), and Theorem 17 holds with \(\mathbf{U} = \begin{bmatrix} 1 \end{bmatrix}\), \(\mathbf{D} = \begin{bmatrix} \sqrt{2} & 0 \end{bmatrix}\), and \(\mathbf{V}\) the orthogonal matrix \(\mathbf{Q}\) from Example 32, which is symmetric, so \({\mathbf{V}}^{\top} = \mathbf{V}\):

\[ \begin{aligned} \mathbf{U}\mathbf{D}{\mathbf{V}}^{\top} &= \begin{bmatrix} 1 \end{bmatrix} \begin{bmatrix} \sqrt{2} & 0 \end{bmatrix} \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(substitute)} \\ &= \begin{bmatrix} \sqrt{2} & 0 \end{bmatrix} \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(multiplying by the } 1 \times 1 \text{ identity changes nothing)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} \sqrt{2} & 0 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(move the scalar to the front)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} \sqrt{2} \cdot 1 + 0 \cdot 1 & \sqrt{2} \cdot 1 + 0 \cdot(-1) \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} \sqrt{2} + 0 & \sqrt{2} + 0 \end{bmatrix} && \text{(multiply)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} \sqrt{2} & \sqrt{2} \end{bmatrix} && \text{(add)} \\ &= \begin{bmatrix} 1 & 1 \end{bmatrix} && \text{(divide by } \sqrt{2} \text{)} \end{aligned} \]

\(\mathbf{U}\) is orthogonal because

\[ \begin{aligned} {\mathbf{U}}^{\top}\mathbf{U} &= \begin{bmatrix} 1 \end{bmatrix} \\ &= \mathbf{I}_1, \end{aligned} \]

and the single singular value is \(\sigma_1 = \sqrt{2}\).

DefDefinition 21 (Singular value decomposition)

Definition 21 (Singular value decomposition) Let \(\mathbf{A}\) be an \(n \times p\) matrix with real entries and \(\operatorname{rank}(\mathbf{A}) = r\). A singular value decomposition [12] (SVD) of \(\mathbf{A}\) is a factorization \(\mathbf{A} = \mathbf{U}\mathbf{D}{\mathbf{V}}^{\top}\) with \(\mathbf{U}\), \(\mathbf{D}\) and \(\mathbf{V}\) as in Theorem 17.

DefDefinition 22 (Singular values)

Definition 22 (Singular values) The numbers \(\sigma_1 \ge \cdots \ge \sigma_r > 0\) on the diagonal of \(\mathbf{D}\) in a singular value decomposition of \(\mathbf{A}\) (Definition 21) are the singular values of \(\mathbf{A}\).

ExmExample 33 (An SVD of a \(3 \times 2\) matrix)

Example 33 (An SVD of a \(3 \times 2\) matrix) Let

\[ \mathbf{A} = \begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix} \qquad \mathbf{U} = \begin{bmatrix} \frac{1}{\sqrt{2}} & 0 & \frac{1}{\sqrt{2}} \\ 0 & 1 & 0 \\ \frac{1}{\sqrt{2}} & 0 & -\frac{1}{\sqrt{2}} \end{bmatrix} \qquad \mathbf{D} = \begin{bmatrix} 2 & 0 \\ 0 & \sqrt{2} \\ 0 & 0 \end{bmatrix} \qquad \mathbf{V} = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} \]

\(\mathbf{V}\) is the orthogonal matrix \(\mathbf{Q}\) from Example 32. \(\mathbf{U}\) is orthogonal too. Entry \((i, j)\) of \({\mathbf{U}}^{\top}\mathbf{U}\) is the dot product of columns \(i\) and \(j\) of \(\mathbf{U}\), and those columns are \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_1 = (\frac{1}{\sqrt{2}}, 0, \frac{1}{\sqrt{2}})\), \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_2 = (0, 1, 0)\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_3 = (\frac{1}{\sqrt{2}}, 0, -\frac{1}{\sqrt{2}})\):

\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_1 \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_1 &= \tfrac{1}{\sqrt{2}} \cdot\tfrac{1}{\sqrt{2}} + 0 \cdot 0 + \tfrac{1}{\sqrt{2}} \cdot\tfrac{1}{\sqrt{2}} && \text{(definition of the dot product)} \\ &= \tfrac{1}{2} + 0 + \tfrac{1}{2} && \text{(multiply)} \\ &= 1 && \text{(add)} \end{aligned} \]

\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_2 \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_2 &= 0 \cdot 0 + 1 \cdot 1 + 0 \cdot 0 && \text{(definition of the dot product)} \\ &= 0 + 1 + 0 && \text{(multiply)} \\ &= 1 && \text{(add)} \end{aligned} \]

\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_3 \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_3 &= \tfrac{1}{\sqrt{2}} \cdot\tfrac{1}{\sqrt{2}} + 0 \cdot 0 + \mathopen{}\left(-\tfrac{1}{\sqrt{2}}\right)\mathclose{} \cdot\mathopen{}\left(-\tfrac{1}{\sqrt{2}}\right)\mathclose{} && \text{(definition of the dot product)} \\ &= \tfrac{1}{2} + 0 + \tfrac{1}{2} && \text{(multiply)} \\ &= 1 && \text{(add)} \end{aligned} \]

\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_1 \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_2 &= \tfrac{1}{\sqrt{2}} \cdot 0 + 0 \cdot 1 + \tfrac{1}{\sqrt{2}} \cdot 0 && \text{(definition of the dot product)} \\ &= 0 + 0 + 0 && \text{(multiply)} \\ &= 0 && \text{(add)} \end{aligned} \]

\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_1 \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_3 &= \tfrac{1}{\sqrt{2}} \cdot\tfrac{1}{\sqrt{2}} + 0 \cdot 0 + \tfrac{1}{\sqrt{2}} \cdot\mathopen{}\left(-\tfrac{1}{\sqrt{2}}\right)\mathclose{} && \text{(definition of the dot product)} \\ &= \tfrac{1}{2} + 0 - \tfrac{1}{2} && \text{(multiply)} \\ &= 0 && \text{(add)} \end{aligned} \]

\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_2 \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{u}}\vphantom{u}}_3 &= 0 \cdot\tfrac{1}{\sqrt{2}} + 1 \cdot 0 + 0 \cdot\mathopen{}\left(-\tfrac{1}{\sqrt{2}}\right)\mathclose{} && \text{(definition of the dot product)} \\ &= 0 + 0 + 0 && \text{(multiply)} \\ &= 0 && \text{(add)} \end{aligned} \]

The dot product is symmetric, so these six values fill in all nine entries, and \({\mathbf{U}}^{\top}\mathbf{U} = \mathbf{I}_3\).

\(\mathbf{V}\) is symmetric, so \({\mathbf{V}}^{\top} = \mathbf{V}\). Then

\[ \begin{aligned} \mathbf{D}{\mathbf{V}}^{\top} &= \begin{bmatrix} 2 & 0 \\ 0 & \sqrt{2} \\ 0 & 0 \end{bmatrix} \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(substitute)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} 2 & 0 \\ 0 & \sqrt{2} \\ 0 & 0 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(move the scalar to the front)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} 2 \cdot 1 + 0 \cdot 1 & 2 \cdot 1 + 0 \cdot(-1) \\ 0 \cdot 1 + \sqrt{2} \cdot 1 & 0 \cdot 1 + \sqrt{2} \cdot(-1) \\ 0 \cdot 1 + 0 \cdot 1 & 0 \cdot 1 + 0 \cdot(-1) \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} 2 + 0 & 2 + 0 \\ 0 + \sqrt{2} & 0 - \sqrt{2} \\ 0 + 0 & 0 + 0 \end{bmatrix} && \text{(multiply)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} 2 & 2 \\ \sqrt{2} & -\sqrt{2} \\ 0 & 0 \end{bmatrix} && \text{(add)} \\ &= \begin{bmatrix} \sqrt{2} & \sqrt{2} \\ 1 & -1 \\ 0 & 0 \end{bmatrix} && \text{(divide by } \sqrt{2} \text{)} \end{aligned} \]

and

\[ \begin{aligned} \mathbf{U}\mathbf{D}{\mathbf{V}}^{\top} &= \begin{bmatrix} \frac{1}{\sqrt{2}} & 0 & \frac{1}{\sqrt{2}} \\ 0 & 1 & 0 \\ \frac{1}{\sqrt{2}} & 0 & -\frac{1}{\sqrt{2}} \end{bmatrix} \begin{bmatrix} \sqrt{2} & \sqrt{2} \\ 1 & -1 \\ 0 & 0 \end{bmatrix} && \text{(substitute)} \\ &= \begin{bmatrix} \frac{1}{\sqrt{2}} \cdot\sqrt{2} + 0 \cdot 1 + \frac{1}{\sqrt{2}} \cdot 0 & \frac{1}{\sqrt{2}} \cdot\sqrt{2} + 0 \cdot(-1) + \frac{1}{\sqrt{2}} \cdot 0 \\ 0 \cdot\sqrt{2} + 1 \cdot 1 + 0 \cdot 0 & 0 \cdot\sqrt{2} + 1 \cdot(-1) + 0 \cdot 0 \\ \frac{1}{\sqrt{2}} \cdot\sqrt{2} + 0 \cdot 1 - \frac{1}{\sqrt{2}} \cdot 0 & \frac{1}{\sqrt{2}} \cdot\sqrt{2} + 0 \cdot(-1) - \frac{1}{\sqrt{2}} \cdot 0 \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \begin{bmatrix} 1 + 0 + 0 & 1 + 0 + 0 \\ 0 + 1 + 0 & 0 - 1 + 0 \\ 1 + 0 - 0 & 1 + 0 - 0 \end{bmatrix} && \text{(multiply)} \\ &= \begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix} && \text{(add)} \end{aligned} \]

which is \(\mathbf{A}\). \(\mathbf{A}\) has rank \(2\), because its two columns are linearly independent, and its singular values are \(\sigma_1 = 2\) and \(\sigma_2 = \sqrt{2}\).

RemarkRemark 17 (An SVD is not unique)

Remark 17 (An SVD is not unique). An SVD is not unique [6, Ch. 12], Examples 12.2 and 12.3, p. 378. For any \(i \le r\), multiplying column \(i\) of both \(\mathbf{U}\) and \(\mathbf{V}\) by \(-1\) gives another one. The singular values do not depend on which SVD is chosen [6, Ch. 12], pp. 371 and 378.

For example, in Example 33, multiplying the first columns of \(\mathbf{U}\) and \(\mathbf{V}\) by \(-1\) gives

\[ \mathbf{U}_1 = \begin{bmatrix} -\frac{1}{\sqrt{2}} & 0 & \frac{1}{\sqrt{2}} \\ 0 & 1 & 0 \\ -\frac{1}{\sqrt{2}} & 0 & -\frac{1}{\sqrt{2}} \end{bmatrix} \qquad \mathbf{V}_1 = \frac{1}{\sqrt{2}} \begin{bmatrix} -1 & 1 \\ -1 & -1 \end{bmatrix} \]

Their columns are the columns of \(\mathbf{U}\) and \(\mathbf{V}\), some multiplied by \(-1\), so they are still orthonormal. With the same \(\mathbf{D}\):

\[ \begin{aligned} \mathbf{D}{\mathbf{V}_1}^{\top} &= \begin{bmatrix} 2 & 0 \\ 0 & \sqrt{2} \\ 0 & 0 \end{bmatrix} {\mathopen{}\left(\frac{1}{\sqrt{2}} \begin{bmatrix} -1 & 1 \\ -1 & -1 \end{bmatrix}\right)\mathclose{}}^{\top} && \text{(substitute)} \\ &= \begin{bmatrix} 2 & 0 \\ 0 & \sqrt{2} \\ 0 & 0 \end{bmatrix} \frac{1}{\sqrt{2}} \begin{bmatrix} -1 & -1 \\ 1 & -1 \end{bmatrix} && \text{(definition of the transpose)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} 2 & 0 \\ 0 & \sqrt{2} \\ 0 & 0 \end{bmatrix} \begin{bmatrix} -1 & -1 \\ 1 & -1 \end{bmatrix} && \text{(move the scalar to the front)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} 2 \cdot(-1) + 0 \cdot 1 & 2 \cdot(-1) + 0 \cdot(-1) \\ 0 \cdot(-1) + \sqrt{2} \cdot 1 & 0 \cdot(-1) + \sqrt{2} \cdot(-1) \\ 0 \cdot(-1) + 0 \cdot 1 & 0 \cdot(-1) + 0 \cdot(-1) \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} -2 + 0 & -2 + 0 \\ 0 + \sqrt{2} & 0 - \sqrt{2} \\ 0 + 0 & 0 + 0 \end{bmatrix} && \text{(multiply)} \\ &= \frac{1}{\sqrt{2}} \begin{bmatrix} -2 & -2 \\ \sqrt{2} & -\sqrt{2} \\ 0 & 0 \end{bmatrix} && \text{(add)} \\ &= \begin{bmatrix} -\sqrt{2} & -\sqrt{2} \\ 1 & -1 \\ 0 & 0 \end{bmatrix} && \text{(divide by } \sqrt{2} \text{)} \end{aligned} \]

and

\[ \begin{aligned} \mathbf{U}_1\mathbf{D}{\mathbf{V}_1}^{\top} &= \begin{bmatrix} -\frac{1}{\sqrt{2}} & 0 & \frac{1}{\sqrt{2}} \\ 0 & 1 & 0 \\ -\frac{1}{\sqrt{2}} & 0 & -\frac{1}{\sqrt{2}} \end{bmatrix} \begin{bmatrix} -\sqrt{2} & -\sqrt{2} \\ 1 & -1 \\ 0 & 0 \end{bmatrix} && \text{(substitute)} \\ &= \begin{bmatrix} -\frac{1}{\sqrt{2}} \cdot\mathopen{}\left(-\sqrt{2}\right)\mathclose{} + 0 \cdot 1 + \frac{1}{\sqrt{2}} \cdot 0 & -\frac{1}{\sqrt{2}} \cdot\mathopen{}\left(-\sqrt{2}\right)\mathclose{} + 0 \cdot(-1) + \frac{1}{\sqrt{2}} \cdot 0 \\ 0 \cdot\mathopen{}\left(-\sqrt{2}\right)\mathclose{} + 1 \cdot 1 + 0 \cdot 0 & 0 \cdot\mathopen{}\left(-\sqrt{2}\right)\mathclose{} + 1 \cdot(-1) + 0 \cdot 0 \\ -\frac{1}{\sqrt{2}} \cdot\mathopen{}\left(-\sqrt{2}\right)\mathclose{} + 0 \cdot 1 - \frac{1}{\sqrt{2}} \cdot 0 & -\frac{1}{\sqrt{2}} \cdot\mathopen{}\left(-\sqrt{2}\right)\mathclose{} + 0 \cdot(-1) - \frac{1}{\sqrt{2}} \cdot 0 \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \begin{bmatrix} 1 + 0 + 0 & 1 + 0 + 0 \\ 0 + 1 + 0 & 0 - 1 + 0 \\ 1 + 0 - 0 & 1 + 0 - 0 \end{bmatrix} && \text{(multiply)} \\ &= \begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix} && \text{(add)} \end{aligned} \]

which is \(\mathbf{A}\) again. Both SVDs have the singular values \(\sigma_1 = 2\) and \(\sigma_2 = \sqrt{2}\).

RemarkRemark 18 (Counting zero singular values)

Remark 18 (Counting zero singular values). Some texts, and R’s svd(), also count \(\min(n, p) - r\) singular values equal to \(0\), so that every \(n \times p\) matrix has \(\min(n, p)\) singular values.

For example, \(\mathbf{B} = \begin{bmatrix} 3 & 0 \\ 0 & 0 \end{bmatrix}\) has rank \(r = 1\). Taking

\[ \begin{aligned} \mathbf{U} &= \mathbf{V} \\ &= \mathbf{I}_2 \end{aligned} \]

and \(\mathbf{D} = \mathbf{B}\) gives an SVD \(\mathbf{B} = \mathbf{I}_2 \mathbf{B} {\mathbf{I}_2}^{\top}\), so by Definition 22, \(\mathbf{B}\) has one singular value, \(\sigma_1 = 3\). R’s svd() reports \(\min(2, 2) = 2\) singular values for \(\mathbf{B}\): \(3\) and \(0\).

LemTheorem 18 (An SVD gives an eigendecomposition of \({\mathbf{A}}^{\top}\mathbf{A}\))

Theorem 18 (An SVD gives an eigendecomposition of \({\mathbf{A}}^{\top}\mathbf{A}\)) Let \(\mathbf{A} = \mathbf{U}\mathbf{D}{\mathbf{V}}^{\top}\) be a singular value decomposition (Definition 21) of an \(n \times p\) matrix \(\mathbf{A}\) with singular values \(\sigma_1, \ldots, \sigma_r\). Then \({\mathbf{A}}^{\top}\mathbf{A}\) is symmetric (Definition 3), and

\[ \underbrace{{\mathbf{A}}^{\top}\mathbf{A}}_{p \times p} = \underbrace{\mathbf{V}}_{p \times p}\, \underbrace{\mathbf{\Lambda}}_{p \times p}\, \underbrace{{\mathbf{V}}^{\top}}_{p \times p} \]

where \(\mathbf{\Lambda} \stackrel{\text{def}}{=}{\mathbf{D}}^{\top}\mathbf{D}\) is the \(p \times p\) diagonal matrix whose \(i\)-th diagonal entry \(\lambda_i\) is \(\sigma_i^2\) for \(i \le r\) and \(0\) for \(i > r\). So \(\mathbf{V}\mathbf{\Lambda}{\mathbf{V}}^{\top}\) is an eigendecomposition (Definition 20) of \({\mathbf{A}}^{\top}\mathbf{A}\): column \(i\) of \(\mathbf{V}\) is an eigenvector of \({\mathbf{A}}^{\top}\mathbf{A}\) for the eigenvalue \(\lambda_i\).

Proof. Symmetry.

\[ \begin{aligned} {\mathopen{}\left({\mathbf{A}}^{\top}\mathbf{A}\right)\mathclose{}}^{\top} &= {\mathbf{A}}^{\top}\,{\mathopen{}\left({\mathbf{A}}^{\top}\right)\mathclose{}}^{\top} && \text{(transpose of a product)} \\ &= {\mathbf{A}}^{\top}\mathbf{A} && \text{(transposing twice changes nothing)} \end{aligned} \]

The first step is Theorem 6 in Matrix Multiplication, and the second follows from Definition 6 in Matrices, which swaps rows and columns, so swapping them again restores \(\mathbf{A}\).

The factorization.

\[ \begin{aligned} {\mathbf{A}}^{\top}\mathbf{A} &= {\mathopen{}\left(\mathbf{U}\mathbf{D}{\mathbf{V}}^{\top}\right)\mathclose{}}^{\top}\,\mathbf{U}\mathbf{D}{\mathbf{V}}^{\top} && \text{(substitute the SVD)} \\ &= {\mathopen{}\left(\mathbf{D}{\mathbf{V}}^{\top}\right)\mathclose{}}^{\top}\,{\mathbf{U}}^{\top}\,\mathbf{U}\mathbf{D}{\mathbf{V}}^{\top} && \text{(transpose of the product of } \mathbf{U} \text{ and } \mathbf{D}{\mathbf{V}}^{\top} \text{)} \\ &= {\mathopen{}\left({\mathbf{V}}^{\top}\right)\mathclose{}}^{\top}\,{\mathbf{D}}^{\top}\,{\mathbf{U}}^{\top}\,\mathbf{U}\mathbf{D}{\mathbf{V}}^{\top} && \text{(transpose of the product of } \mathbf{D} \text{ and } {\mathbf{V}}^{\top} \text{)} \\ &= \mathbf{V}\,{\mathbf{D}}^{\top}\,{\mathbf{U}}^{\top}\,\mathbf{U}\mathbf{D}{\mathbf{V}}^{\top} && \text{(transposing twice changes nothing)} \\ &= \mathbf{V}\,{\mathbf{D}}^{\top}\,\mathopen{}\left({\mathbf{U}}^{\top}\mathbf{U}\right)\mathclose{}\,\mathbf{D}{\mathbf{V}}^{\top} && \text{(regroup; matrix multiplication is associative)} \\ &= \mathbf{V}\,{\mathbf{D}}^{\top}\,\mathbf{I}_n\,\mathbf{D}{\mathbf{V}}^{\top} && \text{(} \mathbf{U} \text{ is orthogonal)} \\ &= \mathbf{V}\,\mathopen{}\left({\mathbf{D}}^{\top}\mathbf{D}\right)\mathclose{}\,{\mathbf{V}}^{\top} && \text{(identity matrix)} \\ &= \mathbf{V}\mathbf{\Lambda}{\mathbf{V}}^{\top} && \text{(definition of } \mathbf{\Lambda} \text{)} \end{aligned} \]

The entries of \(\mathbf{\Lambda}\). Write \(\lambda_{ij}\) for entry \((i, j)\) of \(\mathbf{\Lambda}\), so \(\lambda_i = \lambda_{ii}\). Only the entries \(d_{kk}\) with \(k \le r\) of \(\mathbf{D}\) can be nonzero.

\[ \begin{aligned} \lambda_{ij} &= \sum_{k=1}^{n} ({\mathbf{D}}^{\top})_{ik}\, d_{kj} && \text{(definition of matrix multiplication)} \\ &= \sum_{k=1}^{n} d_{ki}\, d_{kj} && \text{(definition of the transpose)} \end{aligned} \]

The term for \(k\) is nonzero only if \(k = i \le r\) and \(k = j\). So \(\lambda_{ij} = 0\) when \(i \neq j\). On the diagonal, only the term \(k = i\) can be nonzero, and for \(i \le r\)

\[ \begin{aligned} \lambda_i &= \lambda_{ii} && \text{(notation)} \\ &= d_{ii}\, d_{ii} && \text{(the only term that can be nonzero)} \\ &= \sigma_i \cdot\sigma_i && \text{(definition of } \mathbf{D} \text{)} \\ &= \sigma_i^2 && \text{(notation for a square)} \end{aligned} \]

For \(i > r\), every term is \(0\), so \(\lambda_i = 0\).

The eigenvectors. Write \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_i\) for column \(i\) of \(\mathbf{V}\), and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i\) for the indicator vector (Definition 21 in Vectors) of length \(p\). Column \(i\) of \({\mathbf{V}}^{\top}\mathbf{V} = \mathbf{I}_p\) says \({\mathbf{V}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_i = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i\). Then

\[ \begin{aligned} {\mathbf{A}}^{\top}\mathbf{A}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_i &= \mathbf{V}\mathbf{\Lambda}\,{\mathbf{V}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_i && \text{(the factorization)} \\ &= \mathbf{V}\mathbf{\Lambda}\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i && \text{(} {\mathbf{V}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_i = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i \text{)} \\ &= \mathbf{V}\,(\lambda_i\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i) && \text{(column } i \text{ of the diagonal matrix } \mathbf{\Lambda} \text{)} \\ &= \lambda_i\,\mathbf{V}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i && \text{(move the scalar } \lambda_i \text{ to the front)} \\ &= \lambda_i\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_i && \text{(} \mathbf{V}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i \text{ is column } i \text{ of } \mathbf{V} \text{)} \end{aligned} \]

and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_i \neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) because it has norm \(1\).

ExmExample 34 (\({\mathbf{A}}^{\top}\mathbf{A}\) for the matrix of the SVD example)

Example 34 (\({\mathbf{A}}^{\top}\mathbf{A}\) for the matrix of the SVD example) For \(\mathbf{A}\) in Example 33:

\[ \begin{aligned} {\mathbf{A}}^{\top}\mathbf{A} &= \begin{bmatrix} 1 & 1 & 1 \\ 1 & -1 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix} && \text{(definition of the transpose)} \\ &= \begin{bmatrix} 1 + 1 + 1 & 1 - 1 + 1 \\ 1 - 1 + 1 & 1 + 1 + 1 \end{bmatrix} && \text{(definition of matrix multiplication)} \\ &= \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix} && \text{(add)} \end{aligned} \]

Theorem 18 says its eigenvalues are \(\sigma_1^2 = 4\) and \(\sigma_2^2 = 2\), with the columns of \(\mathbf{V}\) as eigenvectors. Checking the first column, up to its factor \(\frac{1}{\sqrt{2}}\):

\[ \begin{aligned} \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix} \begin{bmatrix} 1 \\ 1 \end{bmatrix} &= \begin{bmatrix} 3 \cdot 1 + 1 \cdot 1 \\ 1 \cdot 1 + 3 \cdot 1 \end{bmatrix} && \text{(definition of matrix-vector multiplication)} \\ &= \begin{bmatrix} 3 + 1 \\ 1 + 3 \end{bmatrix} && \text{(multiply)} \\ &= \begin{bmatrix} 4 \\ 4 \end{bmatrix} && \text{(add)} \\ &= 4 \begin{bmatrix} 1 \\ 1 \end{bmatrix} && \text{(factor out } 4 \text{)} \end{aligned} \]

and the second:

\[ \begin{aligned} \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix} \begin{bmatrix} 1 \\ -1 \end{bmatrix} &= \begin{bmatrix} 3 \cdot 1 + 1 \cdot(-1) \\ 1 \cdot 1 + 3 \cdot(-1) \end{bmatrix} && \text{(definition of matrix-vector multiplication)} \\ &= \begin{bmatrix} 3 - 1 \\ 1 - 3 \end{bmatrix} && \text{(multiply)} \\ &= \begin{bmatrix} 2 \\ -2 \end{bmatrix} && \text{(add)} \\ &= 2 \begin{bmatrix} 1 \\ -1 \end{bmatrix} && \text{(factor out } 2 \text{)} \end{aligned} \]

RemarkRemark 19 (Building an SVD from an eigendecomposition)

Remark 19 (Building an SVD from an eigendecomposition). Theorem 18 matches Banerjee and Roy [6, Ch. 12], pp. 372-373, which works in the other direction: it constructs an SVD from a spectral decomposition of \({\mathbf{A}}^{\top}\mathbf{A}\) and sets \(\sigma_i = \sqrt{\lambda_i}\).

For example, in Example 34 the eigenvalues of \({\mathbf{A}}^{\top}\mathbf{A}\) are \(4\) and \(2\), and \(\sqrt{4} = 2\) and \(\sqrt{2}\) are the singular values \(\sigma_1\) and \(\sigma_2\) of \(\mathbf{A}\) from Example 33.

5 Definite Matrices

DefDefinition 23 (Positive semidefinite matrix)

Definition 23 (Positive semidefinite matrix) A \(p \times p\) matrix \(\mathbf{A}\) is positive semidefinite [13] if it satisfies both conditions:

  • \(\mathbf{A}\) is symmetric (Definition 3).
  • Every quadratic form (Definition 13) in \(\mathbf{A}\) is non-negative: \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} \ge 0\) for every vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) of length \(p\).
ExmExample 35 (A positive semidefinite matrix)

Example 35 (A positive semidefinite matrix) Let \(\mathbf{B} = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}\), which is symmetric. For any \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (x_1, x_2)\):

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{B} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= x_1^2 + x_1 x_2 + x_2 x_1 + x_2^2 && \text{(multiply out the quadratic form)} \\ &= (x_1 + x_2)^2 && \text{(complete the square)} \\ &\ge 0 && \text{(a square is non-negative)} \end{aligned} \]

So \(\mathbf{B}\) is positive semidefinite.

LemTheorem 19 (Entries and symmetry of a Gram matrix)

Theorem 19 (Entries and symmetry of a Gram matrix) Let \(\mathbf{A}\) be an \(m \times n\) matrix with columns \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_1, \ldots, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_n\), and let \(\mathbf{G} = {\mathbf{A}}^{\top} \mathbf{A}\) (Definition 6 in Column Space and Null Space).

  1. Entry \((i, j)\) of \(\mathbf{G}\) is \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_i \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_j\).
  2. \(\mathbf{G}\) is symmetric (Definition 3).
  3. \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{G} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} = \mathopen{}\left\lVert\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}^2 \ge 0\) for every \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^n\), so \(\mathbf{G}\) is positive semidefinite (Definition 23).

Proof. Entries. Row \(i\) of \({\mathbf{A}}^{\top}\) is \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_i}^{\top}\), so entry \((i, j)\) of \({\mathbf{A}}^{\top} \mathbf{A}\) is the dot product of that row with column \(j\) of \(\mathbf{A}\), namely \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_i}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_j = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_i \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_j\).

Symmetry. The dot product is commutative, so

\[ \begin{aligned} g_{ij} &= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_i \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_j \\ &= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_j \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{a}}\vphantom{a}}_i \\ &= g_{ji}. \end{aligned} \]

Positive semidefiniteness. For any \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\),

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{G} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\,{\mathbf{A}}^{\top}\,\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(substitute } \mathbf{G} \text{)} \\ &= {(\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})}^{\top}\,(\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) && \text{(}\href{linear-algebra-matrix-multiplication.qmd#thm-transpose-product}{\text{Theorem~6 in Matrix Multiplication}}\text{)} \\ &= \mathopen{}\left\lVert\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}^2 && \text{(}\href{linear-algebra-vectors.qmd#eq-norm-dot-product}{\text{Equation~5 in Vectors}}\text{, squared)} \\ &\ge 0. && \text{(a squared norm is nonnegative)} \end{aligned} \]

ExmExample 36 (Checking the properties on a small Gram matrix)

Example 36 (Checking the properties on a small Gram matrix) For \(\mathbf{A}\) in Example 8 in Column Space and Null Space, \(\mathbf{G} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\). It is symmetric: \(g_{12} = g_{21} = 1\). Take \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, -1)\). Then

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{G} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= \begin{bmatrix} 1 & -1 \end{bmatrix} \begin{bmatrix} 1 \\ -1 \end{bmatrix} && \text{(} \mathbf{G} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, -1) \text{)} \\ &= 2, && \text{(multiply and add)} \end{aligned} \]

and \(\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 0, -1)\) has \(\mathopen{}\left\lVert\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}^2 = 1 + 0 + 1 = 2\), as the theorem says. The converse (every positive semidefinite matrix is a Gram matrix) is this theorem.

CorCorollary 4 (A self outer product is positive semidefinite)

Corollary 4 (A self outer product is positive semidefinite) For a vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\), the self outer product \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\) is positive semidefinite.

Proof. Let \(\mathbf{A} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\), the \(1 \times n\) matrix whose one row is \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\). Then

\[ \begin{aligned} {\mathbf{A}}^{\top} \mathbf{A} &= {\mathbf{A}}^{\top} \mathbf{A} && \text{(definition of the Gram matrix)} \\ &= {\mathopen{}\left({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\right)\mathclose{}}^{\top} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} && \text{(substitute } \mathbf{A} \text{)} \\ &= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} && \text{(the transpose of a transpose is the original matrix)} \\ &= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} && \text{(definition of the self outer product)} \end{aligned} \]

So \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\) is the Gram matrix of \(\mathbf{A}\), and Theorem 19 says that a Gram matrix is positive semidefinite.

ExmExample 37 (A quadratic form in a self outer product)

Example 37 (A quadratic form in a self outer product) For \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 2)\), the example of a self outer product found \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} = \begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix}\), which is symmetric. Take \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} = (2, -1)\). The quadratic form in \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\) at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\) is

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} &= \begin{bmatrix} 2 & -1 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix} \begin{bmatrix} 2 \\ -1 \end{bmatrix} && \text{(definition of the quadratic form)} \\ &= \begin{bmatrix} 2 & -1 \end{bmatrix} \begin{bmatrix} 0 \\ 0 \end{bmatrix} && \text{(} 1 \cdot 2 + 2 \cdot(-1) = 0 \text{ and } 2 \cdot 2 + 4 \cdot(-1) = 0 \text{)} \\ &= 0 && \text{(multiply and add)} \end{aligned} \]

The quadratic form is \(0\) here, which is not negative, as Corollary 4 says. It is not positive either, because \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\) is orthogonal to \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\).

DefDefinition 24 (Positive definite matrix)

Definition 24 (Positive definite matrix) A \(p \times p\) matrix \(\mathbf{A}\) is positive definite [13] if it satisfies both conditions:

  • \(\mathbf{A}\) is symmetric (Definition 3).
  • Every quadratic form (Definition 13) in \(\mathbf{A}\) at a nonzero vector is positive: \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} > 0\) for every vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) of length \(p\).
ExmExample 38 (Positive definite, semidefinite, and neither)

Example 38 (Positive definite, semidefinite, and neither)  

  • The identity matrix \(\mathbf{I}_p\) (Definition 7 in Matrix Multiplication) is positive definite: \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{I}_p \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} = \sum_{i=1}^px_i^2\), which is positive unless every \(x_i\) is \(0\).

  • \(\mathbf{B} = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}\) from Example 35 is positive semidefinite but not positive definite: at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, -1) \neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\),

    \[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{B} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= (1 - 1)^2 \\ &= 0. \end{aligned} \]

  • \(\mathbf{D} = \begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix}\) is symmetric but not positive semidefinite: at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, -1)\),

    \[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{D} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= 1 - 2 - 2 + 1 \\ &= -2 \\ &< 0. \end{aligned} \]

RemarkRemark 20 (Why the definition requires symmetry)

Remark 20 (Why the definition requires symmetry). A positive definite matrix is positive semidefinite (Definition 23): \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} > 0\) for every \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), and \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}}^{\top}\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}= 0\). For example, \(\mathbf{I}_p\) in Example 38 is both.

Some sources drop the symmetry condition from both definitions. These notes keep it, because without it a matrix can pass the quadratic-form condition and still have no real eigenvalues (Definition 18). For example, \(\mathbf{C} = \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix}\) is not symmetric, and for any \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (x_1, x_2)\):

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{C} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= x_1 (x_1 + x_2) + x_2 (-x_1 + x_2) && \text{(multiply out the quadratic form)} \\ &= x_1^2 + x_1 x_2 - x_2 x_1 + x_2^2 && \text{(distribute)} \\ &= x_1^2 + x_2^2 && \text{(the middle terms cancel)} \end{aligned} \]

which is positive for every \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\). But suppose \(\mathbf{C}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} = \lambda\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}\) for a real number \(\lambda\). Matching entries gives \(v_1 + v_2 = \lambda v_1\) and \(-v_1 + v_2 = \lambda v_2\), so \(v_2 = (\lambda- 1)\, v_1\) and \(-v_1 = (\lambda- 1)\, v_2\). Substituting the first into the second gives \(-v_1 = (\lambda- 1)^2 v_1\), so \(\mathopen{}\left(1 + (\lambda- 1)^2\right)\mathclose{} v_1 = 0\). Since \(1 + (\lambda- 1)^2 > 0\), \(v_1 = 0\), and then

\[ \begin{aligned} v_2 &= (\lambda- 1)\, v_1 \\ &= 0. \end{aligned} \]

So no real \(\lambda\) has an eigenvector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).

See also [13].

LemTheorem 20 (Definiteness and eigenvalues)

Theorem 20 (Definiteness and eigenvalues) Let \(\mathbf{A}\) be a \(p \times p\) symmetric matrix with real entries, with eigendecomposition \(\mathbf{A} = \mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top}\) (Definition 20) and eigenvalues \(\lambda_1, \ldots, \lambda_p\) on the diagonal of \(\mathbf{\Lambda}\). Then:

  • \(\mathbf{A}\) is positive semidefinite (Definition 23) if and only if every \(\lambda_i \ge 0\).
  • \(\mathbf{A}\) is positive definite (Definition 24) if and only if every \(\lambda_i > 0\).

Proof. For any vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) of length \(p\), let \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} = {\mathbf{Q}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\). Then:

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(substitute the eigendecomposition)} \\ &= {\mathopen{}\left({\mathbf{Q}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{}}^{\top}\mathbf{\Lambda}\mathopen{}\left({\mathbf{Q}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{} && \text{(transpose of a product)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}}^{\top}\mathbf{\Lambda}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} && \text{(definition of } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} \text{)} \\ &= \sum_{i=1}^p\lambda_i y_i^2 && \text{(} \mathbf{\Lambda} \text{ is diagonal)} \end{aligned} \]

The second step is Theorem 6 in Matrix Multiplication. Also, \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) exactly when \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\): \(\mathbf{Q}{\mathbf{Q}}^{\top} = \mathbf{I}_p\) for an orthogonal matrix (Definition 10), so \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathbf{Q}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\).

If every \(\lambda_i \ge 0\), then every term \(\lambda_i y_i^2 \ge 0\), so \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} \ge 0\). If every \(\lambda_i > 0\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), then some \(y_i \neq 0\), so at least one term is positive and the rest are non-negative, and \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} > 0\).

Conversely, take \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{q}}\vphantom{q}}_i\), column \(i\) of \(\mathbf{Q}\), which is not \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) because it has norm 1. Then \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} = {\mathbf{Q}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{q}}\vphantom{q}}_i\) is column \(i\) of \({\mathbf{Q}}^{\top}\mathbf{Q} = \mathbf{I}_p\), so \(y_i = 1\) and every other entry of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\) is \(0\), and the display gives \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{q}}\vphantom{q}}_i}^{\top}\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{q}}\vphantom{q}}_i = \lambda_i\). So if \(\mathbf{A}\) is positive semidefinite, \(\lambda_i \ge 0\), and if \(\mathbf{A}\) is positive definite, \(\lambda_i > 0\).

ExmExample 39 (Reading definiteness off the eigenvalues)

Example 39 (Reading definiteness off the eigenvalues)  

  • \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\) has eigenvalues \(3\) and \(1\) (Example 29), both positive, so it is positive definite.
  • \(\mathbf{B} = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}\) has eigenvalues \(2\), for the eigenvector \((1, 1)\), and \(0\), for the eigenvector \((1, -1)\), so it is positive semidefinite but not positive definite, as Example 38 found directly.
DefDefinition 25 (Condition number of a positive definite matrix)

Definition 25 (Condition number of a positive definite matrix) Let \(\mathbf{S}\) be a \(p \times p\) positive definite matrix with eigenvalues \(\lambda_{\min} \le \cdots \le \lambda_{\max}\), all positive by definiteness and eigenvalues. The condition number of \(\mathbf{S}\) is

\[ \kappa(\mathbf{S}) \stackrel{\text{def}}{=}\frac{\lambda_{\max}}{\lambda_{\min}}. \]

The condition number is at least \(1\), that is, \(\kappa(\mathbf{S}) \ge 1\), because \(\lambda_{\max} \ge \lambda_{\min} > 0\), and \(\kappa(\mathbf{S}) = 1\) exactly when all eigenvalues coincide.

ExmExample 40 (A condition number of 3)

Example 40 (A condition number of 3) \(\mathbf{S} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\) has eigenvalues \(3\) and \(1\) (eigenvalue example), so \(\kappa(\mathbf{S}) = 3 / 1 = 3\). The identity matrix \(\mathbf{I}_2\) has eigenvalues \(1\) and \(1\), so \(\kappa(\mathbf{I}_2) = 1\).

DefDefinition 26 (Condition number of a matrix)

Definition 26 (Condition number of a matrix) The condition number of an \(n \times p\) matrix \(\mathbf{A}\) is the ratio of its largest to its smallest singular value, counting all \(\min(n, p)\) singular values of \(\mathbf{A}\), zeros included (Remark 18) [14]:

\[ \kappa(\mathbf{A}) \stackrel{\text{def}}{=}\frac{\sigma_{\max}}{\sigma_{\min}} = \frac{\sigma_1}{\sigma_{\min(n,p)}}. \]

This generalizes Definition 25, which covers positive definite matrices. For a nonzero matrix, \(\kappa(\mathbf{A}) \ge 1\). When the columns of a tall matrix \(\mathbf{A}\) (\(n \ge p\)) are linearly dependent, its smallest singular value is \(0\), and \(\kappa(\mathbf{A}) = \infty\).

ExmExample 41 (The condition number of a matrix with nearly parallel columns)

Example 41 (The condition number of a matrix with nearly parallel columns) Let \(\mathbf{A} = \begin{bmatrix} 1 & 1 \\ 1 & 1.01 \end{bmatrix}\). Its columns, \((1, 1)\) and \((1, 1.01)\), are nearly parallel. We find its singular values and their ratio, then repeat after multiplying every entry by \(1000\):

entries_mcn <- c(1, 1, 1, 1.01)
a_mcn <- matrix(entries_mcn, nrow = 2)
svd_mcn <- svd(a_mcn)
sv_mcn <- svd_mcn$d
a_mcn_scaled <- 1000 * a_mcn
svd_mcn_scaled <- svd(a_mcn_scaled)
sv_mcn_scaled <- svd_mcn_scaled$d
sv_mcn
#> [1] 2.0050125 0.0049875
sv_mcn[1] / sv_mcn[2]
#> [1] 402.0075
sv_mcn_scaled[1] / sv_mcn_scaled[2]
#> [1] 402.0075
import numpy as np

a_mcn = np.array([[1.0, 1.0], [1.0, 1.01]])
sv_mcn = np.linalg.svd(a_mcn, compute_uv=False)
sv_mcn_scaled = np.linalg.svd(1000 * a_mcn, compute_uv=False)
print(sv_mcn)
#> [2.0050125 0.0049875]
print(sv_mcn[0] / sv_mcn[1])
#> 402.0075124842948
print(sv_mcn_scaled[0] / sv_mcn_scaled[1])
#> 402.00751248429236

The singular values are 2.005 and 0.005, so \(\kappa(\mathbf{A}) =\) 402. Multiplying every entry by \(1000\) multiplies both singular values by \(1000\), so the condition number is still 402.

RemarkRemark 21 (Other condition numbers)

Remark 21 (Other condition numbers). Other norms give other condition numbers. This definition uses the 2-norm, which is the one built from the singular values. For a positive definite matrix, the singular values are the eigenvalues, and this definition agrees with the ratio of its largest to its smallest eigenvalue in Definition 25.

LemTheorem 21 (Forming \({\mathbf{A}}^{\top}\mathbf{A}\) squares the condition number)

Theorem 21 (Forming \({\mathbf{A}}^{\top}\mathbf{A}\) squares the condition number) Let \(\mathbf{A}\) be a nonzero \(n \times p\) matrix with \(n \ge p\). Then

\[ \kappa({\mathbf{A}}^{\top}\mathbf{A}) = \kappa(\mathbf{A})^2. \]

Proof. By Theorem 18, the eigenvalues of the symmetric matrix \({\mathbf{A}}^{\top}\mathbf{A}\) are \(\sigma_1^2 \ge \cdots \ge \sigma_p^2\), where \(\sigma_1 \ge \cdots \ge \sigma_p\) are the singular values of \(\mathbf{A}\), with zeros counted (Remark 18). These eigenvalues are non-negative, so for the symmetric matrix \({\mathbf{A}}^{\top}\mathbf{A}\) they are also its singular values. Therefore

\[ \kappa({\mathbf{A}}^{\top}\mathbf{A}) = \frac{\sigma_1^2}{\sigma_p^2} = \mathopen{}\left(\frac{\sigma_1}{\sigma_p}\right)\mathclose{}^2 = \kappa(\mathbf{A})^2. \]

If \(\sigma_p = 0\), both sides are \(\infty\).

ExmExample 42 (Squaring the condition number)

Example 42 (Squaring the condition number) For the matrix \(\mathbf{A}\) from Example 41, compare \(\kappa(\mathbf{A})\) with \(\kappa({\mathbf{A}}^{\top}\mathbf{A})\):

# Uses a_mcn and sv_mcn from the earlier example
gram_cng <- crossprod(a_mcn)
kappa_a_cng <- sv_mcn[1] / sv_mcn[2]
svd_gram_cng <- svd(gram_cng)
sv_gram_cng <- svd_gram_cng$d
sv_gram_cng[1] / sv_gram_cng[2]
#> [1] 161610
kappa_a_cng^2
#> [1] 161610
# Uses a_mcn and sv_mcn from the earlier example
gram_cng = a_mcn.T @ a_mcn
kappa_a_cng = sv_mcn[0] / sv_mcn[1]
sv_gram_cng = np.linalg.svd(gram_cng, compute_uv=False)
print(sv_gram_cng[0] / sv_gram_cng[1])
#> 161610.0400912735
print(kappa_a_cng ** 2)
#> 161610.04009381044

The condition number of \({\mathbf{A}}^{\top}\mathbf{A}\) is 161610, and \(\kappa(\mathbf{A})^2 =\) 161610, as Theorem 21 says.

DefDefinition 27 (Ill-conditioned matrix)

Definition 27 (Ill-conditioned matrix) A matrix \(\mathbf{A}\) is ill-conditioned when its condition number is so large that small relative changes in its entries, such as those from rounding, can change the solution to a linear system by a much larger relative amount. The relative change in the solution can be up to \(\kappa(\mathbf{A})\) times the relative change in the entries. A solution computed with \(\kappa(\mathbf{A}) \approx 10^k\) therefore loses about \(k\) of the digits that the floating-point arithmetic carries [15], Lecture 12.

There is no sharp threshold: how large is too large depends on how many accurate digits the answer needs.

DefDefinition 28 (Well-conditioned matrix)

Definition 28 (Well-conditioned matrix) A matrix \(\mathbf{A}\) is well-conditioned when its condition number is close to \(1\), so it is not ill-conditioned.

ExmExample 43 (A small change in the right-hand side moves the solution a long way)

Example 43 (A small change in the right-hand side moves the solution a long way) Let \(\mathbf{A}\) be the matrix from Example 41, and let \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = (2, 2.01)\). The system \(\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) has solution \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 1)\). Change the second entry of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) from \(2.01\) to \(2.011\), and solve again:

# Uses a_mcn and sv_mcn from the previous example
b_ic <- c(2, 2.01)
b_new_ic <- c(2, 2.011)
x_ic <- solve(a_mcn, b_ic)
x_new_ic <- solve(a_mcn, b_new_ic)
x_ic
#> [1] 1 1
x_new_ic
#> [1] 0.9 1.1

# Relative changes, measured with the Euclidean length
db_ic <- (b_new_ic - b_ic)^2 |> sum() |> sqrt()
nb_ic <- b_ic^2 |> sum() |> sqrt()
dx_ic <- (x_new_ic - x_ic)^2 |> sum() |> sqrt()
nx_ic <- x_ic^2 |> sum() |> sqrt()
rel_b_ic <- db_ic / nb_ic
rel_x_ic <- dx_ic / nx_ic
c(rel_b_ic, rel_x_ic, rel_x_ic / rel_b_ic)
#> [1] 3.526706e-04 1.000000e-01 2.835507e+02
# Uses a_mcn and sv_mcn from the previous example
b_ic = np.array([2.0, 2.01])
b_new_ic = np.array([2.0, 2.011])
x_ic = np.linalg.solve(a_mcn, b_ic)
x_new_ic = np.linalg.solve(a_mcn, b_new_ic)
print(x_ic)
#> [1. 1.]
print(x_new_ic)
#> [0.9 1.1]

# Relative changes, measured with the Euclidean length
rel_b_ic = np.linalg.norm(b_new_ic - b_ic) / np.linalg.norm(b_ic)
rel_x_ic = np.linalg.norm(x_new_ic - x_ic) / np.linalg.norm(x_ic)
print(rel_b_ic, rel_x_ic, rel_x_ic / rel_b_ic)
#> 0.0003526706147110207 0.1000000000000334 283.55070093371313

The right-hand side changes by a relative amount of 0.00035, and the solution changes by a relative amount of 0.100, about 284 times as much. That amplification is what Definition 27 describes, and it cannot exceed \(\kappa(\mathbf{A}) =\) 402.0.

LemTheorem 22 (A positive definite matrix has a positive definite inverse)

Theorem 22 (A positive definite matrix has a positive definite inverse) Let \(\mathbf{A}\) be a \(p \times p\) positive definite matrix (Definition 24) with real entries, with eigendecomposition \(\mathbf{A} = \mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top}\) (Definition 20) and eigenvalues \(\lambda_1, \ldots, \lambda_p\). Then \(\mathbf{A}\) is invertible (Definition 6),

\[ \mathbf{A}^{-1} = \mathbf{Q}\,\mathbf{\Lambda}^{-1}\,{\mathbf{Q}}^{\top}, \qquad \mathbf{\Lambda}^{-1} = \begin{bmatrix} 1/\lambda_1 & \cdots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \cdots & 1/\lambda_p \end{bmatrix}, \]

and \(\mathbf{A}^{-1}\) is positive definite.

Proof. By Theorem 20, every \(\lambda_i > 0\), so \(\mathbf{\Lambda}^{-1}\) is well defined, and multiplying the two diagonal matrices entry by entry gives

\[ \begin{aligned} \mathbf{\Lambda}\mathbf{\Lambda}^{-1} &= \mathbf{\Lambda}^{-1}\mathbf{\Lambda} \\ &= \mathbf{I}_p. \end{aligned} \]

Write \(\mathbf{B} = \mathbf{Q}\mathbf{\Lambda}^{-1}{\mathbf{Q}}^{\top}\). Then:

\[ \begin{aligned} \mathbf{A}\mathbf{B} &= \mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top}\mathbf{Q}\mathbf{\Lambda}^{-1}{\mathbf{Q}}^{\top} && \text{(substitute)} \\ &= \mathbf{Q}\mathbf{\Lambda}\mathbf{I}_p\mathbf{\Lambda}^{-1}{\mathbf{Q}}^{\top} && \text{(} {\mathbf{Q}}^{\top}\mathbf{Q} = \mathbf{I}_p \text{)} \\ &= \mathbf{Q}\mathbf{\Lambda}\mathbf{\Lambda}^{-1}{\mathbf{Q}}^{\top} && \text{(identity matrix)} \\ &= \mathbf{Q}{\mathbf{Q}}^{\top} && \text{(} \mathbf{\Lambda}\mathbf{\Lambda}^{-1} = \mathbf{I}_p \text{)} \\ &= \mathbf{I}_p && \text{(} \mathbf{Q} \text{ is orthogonal)} \end{aligned} \]

The same steps, with \(\mathbf{\Lambda}^{-1}\) and \(\mathbf{\Lambda}\) swapped, give \(\mathbf{B}\mathbf{A} = \mathbf{I}_p\), so \(\mathbf{B} = \mathbf{A}^{-1}\) (Definition 5). The steps with \(\mathbf{Q}\) use Definition 10, whose remark records that \(\mathbf{Q}{\mathbf{Q}}^{\top} = \mathbf{I}_p\) too, and the identity step uses Theorem 4 in Matrix Multiplication.

\(\mathbf{A}^{-1}\) is symmetric by Corollary 1. For \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), let \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} = {\mathbf{Q}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\), which is not \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) because \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathbf{Q}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\). As in the proof of Theorem 20:

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\mathbf{A}^{-1}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}}^{\top}\mathbf{\Lambda}^{-1}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} && \text{(substitute } \mathbf{A}^{-1} = \mathbf{Q}\mathbf{\Lambda}^{-1}{\mathbf{Q}}^{\top} \text{)} \\ &= \sum_{i=1}^p\frac{y_i^2}{\lambda_i} && \text{(} \mathbf{\Lambda}^{-1} \text{ is diagonal)} \\ &> 0 && \text{(each } \lambda_i > 0 \text{, and some } y_i \neq 0 \text{)} \end{aligned} \]

So \(\mathbf{A}^{-1}\) is positive definite.

ExmExample 44 (Inverting a positive definite matrix)

Example 44 (Inverting a positive definite matrix) For \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\), Example 32 gives \(\mathbf{Q} = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}\) and eigenvalues \(3\) and \(1\), so:

\[ \begin{aligned} \mathbf{A}^{-1} &= \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} \begin{bmatrix} 1/3 & 0 \\ 0 & 1 \end{bmatrix} \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(substitute; } \mathbf{Q} \text{ is symmetric)} \\ &= \frac{1}{2} \begin{bmatrix} 1/3 & 1 \\ 1/3 & -1 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} && \text{(multiply the first two matrices)} \\ &= \frac{1}{2} \begin{bmatrix} 4/3 & -2/3 \\ -2/3 & 4/3 \end{bmatrix} && \text{(multiply)} \\ &= \frac{1}{3} \begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix} && \text{(simplify)} \end{aligned} \]

Multiplying out,

\[ \begin{aligned} \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \cdot\frac{1}{3}\begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix} &= \frac{1}{3}\begin{bmatrix} 3 & 0 \\ 0 & 3 \end{bmatrix} \\ &= \mathbf{I}_2. \end{aligned} \]

DefDefinition 29 (Paraboloid)

Definition 29 (Paraboloid) A paraboloid [16] in \(\mathbb{R}^{p+1}\) is the graph \({\mathopen{}\left\{(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})) \mid \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\right\}\mathclose{}}\) of a function

\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{S} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + c, \]

where \(\mathbf{S}\) is a nonzero symmetric \(p \times p\) matrix (Definition 3), \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \in \mathbb{R}^p\), and \(c \in \mathbb{R}\). The first term is a quadratic form (Definition 13).

DefDefinition 30 (Elliptic paraboloid)

Definition 30 (Elliptic paraboloid) The paraboloid of Definition 29 is elliptic if \(\mathbf{S}\) is positive definite (Definition 24); it is then a bowl that opens upward.

ExmExample 45 (A bowl with its lowest point at the origin)

Example 45 (A bowl with its lowest point at the origin) Take \(p = 2\), \(\mathbf{S} = \mathbf{I}_2\), \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}_{2 \times 1}\) and \(c = 0\). Then \(\mathbf{S}\) is positive definite (Example 38), so the graph of

\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{I}_2 \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} \\ &= x_1^2 + x_2^2 \end{aligned} \]

is an elliptic paraboloid. At \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 0\); at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 1)\), \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 2\); and \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) > 0\) at every \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), so the origin is the only lowest point of the bowl. Figure 1 shows the surface.

Show code
axis_values <- seq(-2, 2, length.out = 41)
height <- outer(axis_values, axis_values, function(x1, x2) x1^2 + x2^2)

plotly::plot_ly(x = ~axis_values, y = ~axis_values, z = ~t(height)) |>
  plotly::add_surface(showscale = FALSE) |>
  plotly::layout(scene = list(
    xaxis = list(title = "First coordinate, x1"),
    yaxis = list(title = "Second coordinate, x2"),
    zaxis = list(title = "Height, f(x)")
  ))
Figure 1: The surface \(z = x_1^2 + x_2^2\) over \([-2, 2]^2\): a bowl that opens upward, with its lowest point above the origin. Horizontal slices of the bowl are circles around the origin.
DefDefinition 31 (Matrix-induced inner product (\(\mathbf{A}\)-inner product))

Definition 31 (Matrix-induced inner product (\(\mathbf{A}\)-inner product)) Let \(\mathbf{A}\) be a \(p \times p\) symmetric positive definite matrix. The matrix-induced inner product (or \(\mathbf{A}\)-inner product) on \(\mathbb{R}^p\) is the symmetric bilinear form of \(\mathbf{A}\), used as an inner product: for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^p\),

\[ {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rangle\mathclose{}}_{\mathbf{A}} \stackrel{\text{def}}{=}{{B_{\mathbf{A}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right)\mathclose{}}. \]

ExmExample 46 (A matrix-induced inner product calculation)

Example 46 (A matrix-induced inner product calculation) Consider the symmetric matrix

\[ \mathbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}. \]

For any nonzero vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (x_1, x_2) \in \mathbb{R}^2\):

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= 2 x_1^2 + 2 x_1 x_2 + 2 x_2^2 && \text{(expand the quadratic form)} \\ &= (x_1 + x_2)^2 + x_1^2 + x_2^2 && \text{(complete the square)} \end{aligned} \]

which is strictly positive because \(x_1^2 + x_2^2 > 0\) when \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\). Thus \(\mathbf{A}\) is positive definite (Definition 24).

For vectors \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 2)\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= (3, -1)\), their inner product induced by \(\mathbf{A}\) is:

\[ \begin{aligned} {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rangle\mathclose{}}_{\mathbf{A}} &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} && \text{(definition of } {\mathopen{}\left\langle \cdot, \cdot\right\rangle\mathclose{}}_{\mathbf{A}} \text{)} \\ &= \begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \begin{bmatrix} 3 \\ -1 \end{bmatrix} && \text{(substitute vectors and matrix)} \\ &= \begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 2 \cdot 3 + 1 \cdot(-1) \\ 1 \cdot 3 + 2 \cdot(-1) \end{bmatrix} && \text{(}\href{linear-algebra-matrix-multiplication.qmd#def-matvec-mult}{\text{Definition~8 in Matrix Multiplication}}\text{)} \\ &= \begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 5 \\ 1 \end{bmatrix} && \text{(multiply and add)} \\ &= 1 \cdot 5 + 2 \cdot 1 && \text{(}\href{linear-algebra-matrix-multiplication.qmd#def-matvec-mult}{\text{Definition~8 in Matrix Multiplication}}\text{)} \\ &= 7. && \text{(multiply and add)} \end{aligned} \]

In contrast, the standard dot product yields:

\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} &= 1 \cdot 3 + 2 \cdot(-1) \\ &= 1. \end{aligned} \]

The off-diagonal entries

\[ \begin{aligned} A_{12} &= A_{21} \\ &= 1 \end{aligned} \]

couple the coordinate directions, altering lengths and angles from their Euclidean values. When \(\mathbf{A} = \mathbf{I}_p\), the matrix-induced inner product recovers the standard dot product \({\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rangle\mathclose{}}_{\mathbf{I}_p} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\).

RemarkRemark 22 (A symmetric bilinear form needs a positive definite matrix to be an inner product)

Remark 22 (A symmetric bilinear form needs a positive definite matrix to be an inner product). Definition 31 uses the symmetric bilinear form of \(\mathbf{A}\) as an inner product only when \(\mathbf{A}\) is also positive definite. A symmetric matrix \(\mathbf{A}\) that is not positive definite has a nonzero vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) with \({{B_{\mathbf{A}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{}} \le 0\). If \({{B_{\mathbf{A}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{}} < 0\), the form breaks positivity; if \({{B_{\mathbf{A}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{}} = 0\), it breaks definiteness. Either way, that form is not an inner product. For example, take \(\mathbf{S}\) from Example 18 and the vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}} = {(1, -1)}^{\top}\):

\[ \begin{aligned} {{B_{\mathbf{S}}}\mathopen{}\left(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}}\right)\mathclose{}} &= \begin{bmatrix} 1 & -1 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix} \begin{bmatrix} 1 \\ -1 \end{bmatrix} && \text{(}\href{#def-bilinear-form}{\text{Definition~11}}\text{)} \\ &= \begin{bmatrix} 1 & -1 \end{bmatrix} \begin{bmatrix} 1 \cdot 1 + 2 \cdot(-1) \\ 2 \cdot 1 + 1 \cdot(-1) \end{bmatrix} && \text{(multiply the matrix by } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}} \text{)} \\ &= \begin{bmatrix} 1 & -1 \end{bmatrix} \begin{bmatrix} -1 \\ 1 \end{bmatrix} && \text{(multiply and add)} \\ &= 1 \cdot(-1) + (-1) \cdot 1 && \text{(multiply the row by the column)} \\ &= -2. && \text{(multiply and add)} \end{aligned} \]

The quadratic form of \(\mathbf{S}\) is negative at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}}\), so \(\mathbf{S}\) is not positive definite.

LemTheorem 23 (Properties of the matrix-induced inner product)

Theorem 23 (Properties of the matrix-induced inner product) For every \(p \times p\) symmetric positive definite matrix \(\mathbf{A}\), the function \({\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rangle\mathclose{}}_{\mathbf{A}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\) satisfies the four inner product axioms (Definition 5 in Inner Products and Orthogonality): positivity, definiteness, linearity in the first slot, and symmetry. Thus \((\mathbb{R}^p, {\mathopen{}\left\langle \cdot, \cdot\right\rangle\mathclose{}}_{\mathbf{A}})\) is an inner product space (Definition 6 in Inner Products and Orthogonality).

Taking \(\mathbf{A} = \mathbf{I}_p\) recovers the standard dot product (Example 1 in Inner Products and Orthogonality), while taking \(\mathbf{A} = \operatorname{diag}(c_1, \ldots, c_p)\) recovers the weighted inner product (Example 3 in Inner Products and Orthogonality).

Conversely, every inner product on \(\mathbb{R}^p\) has this form for a unique symmetric positive definite matrix \(\mathbf{A}\), whose \((i, j)\) entry is \(A_{ij} = {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\right\rangle\mathclose{}}\), where \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_1, \ldots, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_p\) are the standard basis vectors (Definition 21 in Vectors).

Proof.

  • positivity: \({\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rangle\mathclose{}}_{\mathbf{A}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} \ge 0\) for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\), because \(\mathbf{A}\) is positive definite (Definition 24);
  • definiteness: \({\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rangle\mathclose{}}_{\mathbf{A}} = 0\) if and only if \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}_{p \times 1}\), by definition of a positive definite matrix (Definition 24);
  • linearity in the first slot: for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}} \in \mathbb{R}^p\) and real numbers \(a, b\), \[ \begin{aligned} {\mathopen{}\left\langle a\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ b\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}}\right\rangle\mathclose{}}_{\mathbf{A}} &= {(a\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ b\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})}^{\top}\,\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}} && \text{(definition of } {\mathopen{}\left\langle \cdot, \cdot\right\rangle\mathclose{}}_{\mathbf{A}} \text{)} \\ &= (a\,{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} + b\,{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}}^{\top})\,\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}} && \text{(}\href{linear-algebra-matrix-multiplication.qmd#thm-transpose-sum}{\text{Theorem~5 in Matrix Multiplication}}\text{, }\href{linear-algebra-matrices.qmd#def-scalar-mult}{\text{Definition~9 in Matrices}}\text{, }\href{linear-algebra-matrices.qmd#def-matrix-transpose}{\text{Definition~6 in Matrices}}\text{)} \\ &= a\,{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}} + b\,{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}}^{\top}\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}} && \text{(}\href{linear-algebra-matrix-multiplication.qmd#thm-matmul-distrib}{\text{Theorem~2 in Matrix Multiplication}}\text{)} \\ &= a\,{\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}}\right\rangle\mathclose{}}_{\mathbf{A}} + b\,{\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}}\right\rangle\mathclose{}}_{\mathbf{A}} && \text{(definition of } {\mathopen{}\left\langle \cdot, \cdot\right\rangle\mathclose{}}_{\mathbf{A}} \text{)} \end{aligned} \]
  • symmetry: because \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}}^{\top}\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) is a \(1 \times 1\) scalar and \(\mathbf{A}\) is symmetric (\({\mathbf{A}}^{\top} = \mathbf{A}\)), \[ \begin{aligned} {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rangle\mathclose{}}_{\mathbf{A}} &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}}^{\top}\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(definition of } {\mathopen{}\left\langle \cdot, \cdot\right\rangle\mathclose{}}_{\mathbf{A}} \text{)} \\ &= {\mathopen{}\left({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}}^{\top}\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right)\mathclose{}}^{\top} && \text{(a scalar equals its transpose)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}{\mathbf{A}}^{\top}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} && \text{(}\href{linear-algebra-matrix-multiplication.qmd#thm-transpose-product}{\text{Theorem~6 in Matrix Multiplication}}\text{, twice)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}} && \text{($\mathbf{A}$ is symmetric)} \\ &= {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rangle\mathclose{}}_{\mathbf{A}} && \text{(definition of } {\mathopen{}\left\langle \cdot, \cdot\right\rangle\mathclose{}}_{\mathbf{A}} \text{)} \end{aligned} \]

For the converse, let \({\mathopen{}\left\langle \cdot, \cdot\right\rangle\mathclose{}}\) be any inner product on \(\mathbb{R}^p\). Any vectors \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^p\) can be written in the standard basis as \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \sum_{i=1}^px_i\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= \sum_{j=1}^py_j\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\) (Definition 6 in Subspaces and Rank, Definition 21 in Vectors). By linearity in both slots (Remark 1 in Inner Products and Orthogonality):

\[ \begin{aligned} {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rangle\mathclose{}} &= {\mathopen{}\left\langle \sum_{i=1}^px_i\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i, \sum_{j=1}^py_j\,\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\right\rangle\mathclose{}} && \text{(expand in the standard basis)} \\ &= \sum_{i=1}^p\sum_{j=1}^px_i y_j\,{\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\right\rangle\mathclose{}} && \text{(linearity in both slots)} \\ &= \sum_{i=1}^px_i \sum_{j=1}^pA_{ij} y_j && \text{(define } A_{ij} \stackrel{\text{def}}{=}{\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\right\rangle\mathclose{}} \text{)} \\ &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}. && \text{(}\href{linear-algebra-matrix-multiplication.qmd#def-matvec-mult}{\text{Definition~8 in Matrix Multiplication}}\text{)} \end{aligned} \]

Symmetry of \({\mathopen{}\left\langle \cdot, \cdot\right\rangle\mathclose{}}\) ensures

\[ \begin{aligned} A_{ji} &= {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i\right\rangle\mathclose{}} \\ &= {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\right\rangle\mathclose{}} \\ &= A_{ij}, \end{aligned} \]

so \(\mathbf{A}\) is symmetric (Definition 3). For any \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} = {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rangle\mathclose{}} > 0\) by positivity and definiteness of the inner product, so \(\mathbf{A}\) is positive definite (Definition 24).

For uniqueness, if another matrix \(\mathbf{B}\) satisfies \({\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rangle\mathclose{}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top}\mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\) for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\), evaluating at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\) gives

\[ \begin{aligned} B_{ij} &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i}^{\top}\mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j \\ &= {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\right\rangle\mathclose{}} \\ &= A_{ij}, \end{aligned} \]

so \(\mathbf{B} = \mathbf{A}\).

ExmExample 47 (Checking the inner product axioms, and recovering the matrix from the inner product)

Example 47 (Checking the inner product axioms, and recovering the matrix from the inner product) Take \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\) from Example 46, the vectors \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 2)\), \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= (3, -1)\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{z}}\vphantom{z}} = (0, 4)\), and the numbers \(a = 2\) and \(b = -3\). The code computes both sides of each axiom in Theorem 23, and then the numbers \({\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\right\rangle\mathclose{}}_{\mathbf{A}}\) for the standard basis vectors.

A <- matrix(c(2, 1, 1, 2), nrow = 2)
ip <- function(x, y) drop(t(x) %*% A %*% y)

x <- c(1, 2)
y <- c(3, -1)
z <- c(0, 4)
a <- 2
b <- -3

c(
  positivity = ip(x, x),
  symmetry_xy = ip(x, y),
  symmetry_yx = ip(y, x),
  linearity_left = ip(a * x + b * y, z),
  linearity_right = a * ip(x, z) + b * ip(y, z)
)
#>      positivity     symmetry_xy     symmetry_yx  linearity_left linearity_right 
#>              14               7               7              28              28

e <- diag(2)
recovered <- outer(1:2, 1:2, Vectorize(function(i, j) ip(e[, i], e[, j])))
recovered
#>      [,1] [,2]
#> [1,]    2    1
#> [2,]    1    2

The inner product of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) with itself is 14, which is positive. The two orders give the same number, 7 and 7. Both sides of the linearity axiom equal 28. The matrix of numbers \({\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_i, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\right\rangle\mathclose{}}_{\mathbf{A}}\) is equal to \(\mathbf{A}\), as the converse in Theorem 23 says.

DefDefinition 32 (Matrix-induced norm, \(\mathbf{A}\)-norm)

Definition 32 (Matrix-induced norm, \(\mathbf{A}\)-norm) Let \(\mathbf{A}\) be a \(p \times p\) positive definite matrix. The matrix-induced norm, or \(\mathbf{A}\)-norm, on \(\mathbb{R}^p\) is the induced norm of the matrix-induced inner product \({\mathopen{}\left\langle \cdot, \cdot\right\rangle\mathclose{}}_{\mathbf{A}}\): for every \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\),

\[ \mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}_{\mathbf{A}} \stackrel{\text{def}}{=}\sqrt{{\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rangle\mathclose{}}_{\mathbf{A}}}. \]

ExmExample 48 (A matrix-induced norm calculation)

Example 48 (A matrix-induced norm calculation) Take the positive definite matrix \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\) of Example 46 and the vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 2)\). Then

\[ \begin{aligned} \mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}_{\mathbf{A}} &= \sqrt{{\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rangle\mathclose{}}_{\mathbf{A}}} && \text{(}\href{#def-matrix-induced-norm}{\text{Definition~32}}\text{)} \\ &= \sqrt{{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}} && \text{(}\href{#def-matrix-induced-inner-product}{\text{Definition~31}}\text{)} \\ &= \sqrt{\begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix}} && \text{(substitute the vector and the matrix)} \\ &= \sqrt{\begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 2 \cdot 1 + 1 \cdot 2 \\ 1 \cdot 1 + 2 \cdot 2 \end{bmatrix}} && \text{(}\href{linear-algebra-matrix-multiplication.qmd#def-matvec-mult}{\text{Definition~8 in Matrix Multiplication}}\text{)} \\ &= \sqrt{\begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 4 \\ 5 \end{bmatrix}} && \text{(multiply and add)} \\ &= \sqrt{1 \cdot 4 + 2 \cdot 5} && \text{(}\href{linear-algebra-matrix-multiplication.qmd#def-matvec-mult}{\text{Definition~8 in Matrix Multiplication}}\text{)} \\ &= \sqrt{14}. && \text{(multiply and add)} \end{aligned} \]

The Euclidean norm of the same vector is \(\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{} = \sqrt{1^2 + 2^2} = \sqrt{5}\), so the \(\mathbf{A}\)-norm of this vector is larger than its Euclidean norm.

ExmExample 49 (A symmetric matrix that induces no norm)

Example 49 (A symmetric matrix that induces no norm) The symmetric matrix \(\mathbf{B} = \operatorname{diag}(1, -1)\) is not positive definite, so it has no matrix-induced norm. The square root \(\sqrt{{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{B} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}\) is undefined at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (0, 1)\):

\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{B} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} &= 1 \cdot 0^2 + (-1) \cdot 1^2 && \text{(expand the quadratic form)} \\ &= -1, && \text{(multiply and add)} \end{aligned} \]

and \(-1\) has no square root in \(\mathbb{R}\).

CorCorollary 5 (A positive definite matrix is a transpose product)

Corollary 5 (A positive definite matrix is a transpose product) Let \(\mathbf{A}\) be a \(p \times p\) symmetric positive definite matrix (Definition 24). Then \(\mathbf{A} = {\mathbf{W}}^{\top}\mathbf{W}\) for some invertible \(p \times p\) matrix \(\mathbf{W}\).

Proof. By the spectral theorem (Theorem 15), \(\mathbf{A} = \mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top}\) with \(\mathbf{Q}\) orthogonal and \(\mathbf{\Lambda}\) diagonal with the eigenvalues \(\lambda_i\) of \(\mathbf{A}\) on its diagonal. These are positive (Theorem 20), so \(\mathbf{\Lambda}^{1/2}\), the diagonal matrix with entries \(\sqrt{\lambda_i}\), is defined. Take \(\mathbf{W} = \mathbf{\Lambda}^{1/2}{\mathbf{Q}}^{\top}\). Since \(\mathbf{\Lambda}^{1/2}\) is diagonal, \({(\mathbf{\Lambda}^{1/2})}^{\top} = \mathbf{\Lambda}^{1/2}\), and

\[ \begin{aligned} {\mathbf{W}}^{\top}\mathbf{W} &= \mathbf{Q}\mathbf{\Lambda}^{1/2}\mathbf{\Lambda}^{1/2}{\mathbf{Q}}^{\top} \\ &= \mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top} \\ &= \mathbf{A}. \end{aligned} \]

\(\mathbf{W}\) is invertible, because \({\mathbf{Q}}^{\top}\) has inverse \(\mathbf{Q}\) and \(\mathbf{\Lambda}^{1/2}\) is diagonal with positive entries, so it has the diagonal inverse with entries \(1/\sqrt{\lambda_i}\), and a product of invertible matrices is invertible (Theorem 1).

LemTheorem 24 (A matrix-induced inner product is a dot product of transformed vectors)

Theorem 24 (A matrix-induced inner product is a dot product of transformed vectors) Let \(\mathbf{A}\) be a \(p \times p\) symmetric positive definite matrix, and let \(\mathbf{W}\) be any \(p \times p\) matrix with \(\mathbf{A} = {\mathbf{W}}^{\top}\mathbf{W}\). Then for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^p\), the \(\mathbf{A}\)-inner product (Definition 31) satisfies

\[ {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rangle\mathclose{}}_{\mathbf{A}} = (\mathbf{W}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) \cdot (\mathbf{W}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}). \]

Proof. \[ \begin{aligned} {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rangle\mathclose{}}_{\mathbf{A}} &= {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} {\mathbf{W}}^{\top}\mathbf{W} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\\ &= {(\mathbf{W}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})}^{\top} (\mathbf{W}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) \\ &= (\mathbf{W}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) \cdot (\mathbf{W}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}). \end{aligned} \]

RemarkRemark 23 (Inner product spaces in machine learning)

Remark 23 (Inner product spaces in machine learning). In machine learning and statistics, choosing or learning an inner product space tailors geometric notions of distance, angle, and projection to the structure of the data:

  1. Mahalanobis metric and whitening: Standard Euclidean distance treats all coordinate axes identically, ignoring feature correlations and variance differences. If data features have a symmetric positive definite covariance matrix \(\mathbf{\Sigma}\) (Definition 24), the precision matrix \(\mathbf{A} = \mathbf{\Sigma}^{-1}\) (Theorem 22) defines the Mahalanobis inner product \({\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rangle\mathclose{}}_{\mathbf{\Sigma}^{-1}} = {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\top} \mathbf{\Sigma}^{-1} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\). The induced distance \(d_{\mathbf{\Sigma}^{-1}}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) = \sqrt{{(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}- \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})}^{\top}\mathbf{\Sigma}^{-1}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}- \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})}\) measures statistical distance normalized by variance along each principal direction. Factoring \(\mathbf{\Sigma}^{-1} = {\mathbf{W}}^{\top}\mathbf{W}\) (via Cholesky or spectral decomposition, Corollary 5), this inner product is the Euclidean dot product of the whitened vectors \(\mathbf{W}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and \(\mathbf{W}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\) (Theorem 24).

  2. The kernel trick and reproducing kernel Hilbert spaces (RKHS): Algorithms such as support vector machines (SVMs), kernel ridge regression, and Gaussian processes depend on training vectors solely through pairwise inner products \({\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}_i, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}_j\right\rangle\mathclose{}}\). By mapping data into a higher- or infinite-dimensional inner product space via a feature map \(D: \mathbb{R}^p \to \mathcal{H}\), the kernel trick replaces the computation of \(D(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}_i)\) with an evaluation of a positive definite kernel function: \[ k(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}_i, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}_j) \stackrel{\text{def}}{=}{\mathopen{}\left\langle D(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}_i), D(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}_j)\right\rangle\mathclose{}}_{\mathcal{H}}. \] Mercer’s theorem guarantees that every symmetric positive definite kernel function implicitly defines a valid inner product space \(\mathcal{H}\) (a reproducing kernel Hilbert space) without requiring explicit coordinate representations.

  3. Embeddings and cosine similarity: In natural language processing and recommendation systems, data items are embedded as vectors in high-dimensional representations. The angle between embeddings captures semantic similarity through cosine similarity (Definition 14 in Inner Products and Orthogonality): \[ \operatorname{cossim}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) = \frac{{\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rangle\mathclose{}}}{\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{}\,\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rVert\mathclose{}}. \] When embedding vectors are normalized to unit Euclidean norm (\(\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\right\rVert\mathclose{} = 1\) and \(\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rVert\mathclose{} = 1\)), cosine similarity equals the inner product \({\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\right\rangle\mathclose{}}\) (Corollary 2 in Inner Products and Orthogonality). Similarly, the self-attention mechanism in Transformers computes query-key alignment scores through scaled inner products \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{q}}\vphantom{q}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{k}}\vphantom{k}} / \sqrt{d_k}\).

  4. Weighted and generalized least squares: In linear regression \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\approx \mathbf{X}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}\), when observation errors have unequal variances or known correlations with positive definite covariance matrix \(\mathbf{\Omega}\), ordinary least squares is inefficient. Generalized least squares minimizes the matrix-induced norm of the residual, with \(\mathbf{W} = \mathbf{\Omega}^{-1}\): \[ \min_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}} \mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}- \mathbf{X}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}\right\rVert\mathclose{}_{\mathbf{W}}^2 = \min_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}} {\mathopen{}\left\langle \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}- \mathbf{X}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}- \mathbf{X}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{\beta}}\vphantom{\beta}}\right\rangle\mathclose{}}_{\mathbf{W}}. \]

6 Determinants

DefDefinition 33 (Determinant)

Definition 33 (Determinant) The determinant [17] of a \(p \times p\) matrix \(\mathbf{A}\) with entries \(a_{ij}\), written \(\det(\mathbf{A})\) or \(\mathopen{}\left|\mathbf{A}\right|\mathclose{}\), is the number defined recursively in \(p\):

  • For \(p = 1\), \(\det(\mathbf{A}) = a_{11}\).
  • For \(p \ge 2\), \[ \det(\mathbf{A}) = \sum_{j=1}^p(-1)^{1+j}\, a_{1j} {\det\mathopen{}\left(\mathbf{A}_{(1j)}\right)\mathclose{}}, \] where \(\mathbf{A}_{(1j)}\) is the \((p-1) \times (p-1)\) matrix left after deleting row \(1\) and column \(j\) of \(\mathbf{A}\).
ExmExample 50 (Determinant of a \(2 \times 2\) matrix)

Example 50 (Determinant of a \(2 \times 2\) matrix) For \(\mathbf{A} = \begin{bmatrix} a & b \\ c & d \end{bmatrix}\), deleting row 1 and column 1 leaves \(\begin{bmatrix} d \end{bmatrix}\), and deleting row 1 and column 2 leaves \(\begin{bmatrix} c \end{bmatrix}\). So:

\[ \begin{aligned} \det(\mathbf{A}) &= (-1)^{1+1}\, a {\det\mathopen{}\left(\begin{bmatrix} d \end{bmatrix}\right)\mathclose{}} + (-1)^{1+2}\, b {\det\mathopen{}\left(\begin{bmatrix} c \end{bmatrix}\right)\mathclose{}} && \text{(definition, } p = 2 \text{)} \\ &= a d - b c && \text{(definition, } p = 1 \text{)} \end{aligned} \]

For example,

\[ \begin{aligned} {\det\mathopen{}\left(\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\right)\mathclose{}} &= 2 \cdot 2 - 1 \cdot 1 \\ &= 3. \end{aligned} \]

RemarkRemark 24 (Cofactor expansion)

Remark 24 (Cofactor expansion). The recursive formula in Definition 33 is the cofactor expansion along the first row. Other sources define the determinant as a sum over all orderings of the columns and derive this expansion from it; the two definitions agree [6].

Each ordering picks one entry from each row, from the column that the ordering assigns to that row, and multiplies them. Its sign is \(+1\) if the ordering takes an even number of swaps of two columns to reach from \((1, 2, \ldots, p)\), and \(-1\) if it takes an odd number.

For \(p = 2\), the columns have two orderings. The ordering \((1, 2)\) keeps each column in place, picks the entries \(a_{11}\) and \(a_{22}\), and has sign \(+1\). The ordering \((2, 1)\) swaps the two columns, picks the entries \(a_{12}\) and \(a_{21}\), and has sign \(-1\). The sum \(a_{11} a_{22} - a_{12} a_{21}\) is the value \(a d - b c\) from Example 50.

See also [17].

TipVideo

Chapter 6 of the Essence of linear algebra series by 3Blue1Brown shows the geometric meaning of the determinant:

  • The determinant shows how the determinant measures the factor by which a linear transformation scales areas and volumes, and why a zero determinant corresponds to compressing space into a lower dimension.
LemTheorem 25 (Determinant of a diagonal matrix)

Theorem 25 (Determinant of a diagonal matrix) The determinant (Definition 33) of a \(p \times p\) diagonal matrix (Definition 4) is the product of its diagonal entries:

\[ {\det\mathopen{}\left( \begin{bmatrix} d_1 & \cdots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \cdots & d_p \end{bmatrix} \right)\mathclose{}} = \prod_{i=1}^p d_i \]

Proof. By induction on \(p\). For \(p = 1\), the matrix is \(\begin{bmatrix} d_1 \end{bmatrix}\), and its determinant is \(d_1\) by definition.

For \(p \ge 2\), suppose the result holds for \((p-1) \times (p-1)\) diagonal matrices, and let \(\mathbf{D}\) be \(p \times p\) and diagonal. Row 1 of \(\mathbf{D}\) is \((d_1, 0, \ldots, 0)\), so only the \(j = 1\) term of the definition can be nonzero, and deleting row 1 and column 1 of \(\mathbf{D}\) leaves the diagonal matrix with diagonal entries \(d_2, \ldots, d_p\):

\[ \begin{aligned} \det(\mathbf{D}) &= (-1)^{1+1}\, d_1 {\det\mathopen{}\left(\mathbf{D}_{(11)}\right)\mathclose{}} + \sum_{j=2}^p (-1)^{1+j} \cdot 0 \cdot{\det\mathopen{}\left(\mathbf{D}_{(1j)}\right)\mathclose{}} && \text{(definition)} \\ &= d_1 {\det\mathopen{}\left(\mathbf{D}_{(11)}\right)\mathclose{}} && \text{(drop the zero terms)} \\ &= d_1 \prod_{i=2}^p d_i && \text{(induction hypothesis)} \\ &= \prod_{i=1}^p d_i && \text{(combine)} \end{aligned} \]

ExmExample 51 (Determinant of a scaled identity matrix)

Example 51 (Determinant of a scaled identity matrix) For a number \(c\), \(c\,\mathbf{I}_p\) is diagonal with every diagonal entry equal to \(c\), so \(\det(c\,\mathbf{I}_p) = c^p\). In particular, \(\det(\mathbf{I}_p) = 1\).

LemTheorem 26 (Determinant of a product)

Theorem 26 (Determinant of a product) For \(p \times p\) matrices \(\mathbf{A}\) and \(\mathbf{B}\),

\[ \det(\mathbf{A}\mathbf{B}) = \det(\mathbf{A}) \det(\mathbf{B}). \]

ExmExample 52 (Checking the product rule)

Example 52 (Checking the product rule) For \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 0 & 1 \end{bmatrix}\) and \(\mathbf{B} = \begin{bmatrix} 1 & 0 \\ 3 & 1 \end{bmatrix}\), Example 50 gives

\[ \begin{aligned} \det(\mathbf{A}) &= 2 \cdot 1 - 1 \cdot 0 \\ &= 2 \end{aligned} \]

and

\[ \begin{aligned} \det(\mathbf{B}) &= 1 \cdot 1 - 0 \cdot 3 \\ &= 1. \end{aligned} \]

The product is

\[ \begin{aligned} \mathbf{A}\mathbf{B} &= \begin{bmatrix} 2 \cdot 1 + 1 \cdot 3 & 2 \cdot 0 + 1 \cdot 1 \\ 0 \cdot 1 + 1 \cdot 3 & 0 \cdot 0 + 1 \cdot 1 \end{bmatrix} \\ &= \begin{bmatrix} 5 & 1 \\ 3 & 1 \end{bmatrix}, \end{aligned} \]

with

\[ \begin{aligned} \det(\mathbf{A}\mathbf{B}) &= 5 \cdot 1 - 1 \cdot 3 \\ &= 2 \\ &= \det(\mathbf{A})\det(\mathbf{B}). \end{aligned} \]

RemarkRemark 25 (An example is not a proof)

Remark 25 (An example is not a proof). Example 52 checks Theorem 26 for one pair of \(2 \times 2\) matrices, which shows the theorem holds there but does not prove it for every pair. The proof needs properties of the determinant that these notes don’t develop; see Banerjee and Roy [6].

See also [17].

LemTheorem 27 (The determinant of a symmetric matrix is the product of its eigenvalues)

Theorem 27 (The determinant of a symmetric matrix is the product of its eigenvalues) Let \(\mathbf{A}\) be a \(p \times p\) symmetric matrix with real entries, with eigendecomposition \(\mathbf{A} = \mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top}\) (Definition 20) and eigenvalues \(\lambda_1, \ldots, \lambda_p\) on the diagonal of \(\mathbf{\Lambda}\). Then

\[ \det(\mathbf{A}) = \prod_{i=1}^p \lambda_i. \]

Proof. First, \(\det({\mathbf{Q}}^{\top})\det(\mathbf{Q}) = 1\):

\[ \begin{aligned} \det({\mathbf{Q}}^{\top})\det(\mathbf{Q}) &= \det({\mathbf{Q}}^{\top}\mathbf{Q}) && \text{(determinant of a product)} \\ &= \det(\mathbf{I}_p) && \text{(} \mathbf{Q} \text{ is orthogonal)} \\ &= 1 && \text{(determinant of the identity)} \end{aligned} \]

Then:

\[ \begin{aligned} \det(\mathbf{A}) &= \det(\mathbf{Q}\mathbf{\Lambda}{\mathbf{Q}}^{\top}) && \text{(substitute the eigendecomposition)} \\ &= \det(\mathbf{Q}) \det(\mathbf{\Lambda}) \det({\mathbf{Q}}^{\top}) && \text{(determinant of a product, twice)} \\ &= \det(\mathbf{\Lambda}) \cdot\det({\mathbf{Q}}^{\top})\det(\mathbf{Q}) && \text{(reorder the three numbers)} \\ &= \det(\mathbf{\Lambda}) && \text{(the first display)} \\ &= \prod_{i=1}^p \lambda_i && \text{(determinant of a diagonal matrix)} \end{aligned} \]

The product steps are Theorem 26, the orthogonality step is Definition 10, and the identity and diagonal steps are Theorem 25 and Example 51.

CorCorollary 6 (A positive definite matrix has a positive determinant)

Corollary 6 (A positive definite matrix has a positive determinant) If \(\mathbf{A}\) is positive definite (Definition 24), then \(\det(\mathbf{A}) > 0\).

Proof. By Theorem 20, every eigenvalue of \(\mathbf{A}\) is positive, so their product, which is \(\det(\mathbf{A})\) by Theorem 27, is positive.

ExmExample 53 (Two ways to the same determinant)

Example 53 (Two ways to the same determinant) \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\) has eigenvalues \(3\) and \(1\) (Example 29), so Theorem 27 gives

\[ \begin{aligned} \det(\mathbf{A}) &= 3 \cdot 1 \\ &= 3, \end{aligned} \]

matching \(2 \cdot 2 - 1 \cdot 1 = 3\) from Example 50.

Definitions on this page

Back to top

References

[1]
Wikipedia contributors, “Symmetric matrix — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Symmetric_matrix
[2]
Wikipedia contributors, “Diagonal matrix — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Diagonal_matrix
[3]
Wikipedia contributors, “Invertible matrix — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Invertible_matrix
[4]
Wikipedia contributors, “Idempotent matrix — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Idempotent_matrix
[5]
Wikipedia contributors, “Orthogonal matrix — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Orthogonal_matrix
[6]
S. Banerjee and A. Roy, Linear algebra and matrix analysis for statistics, vol. 181. Crc Press Boca Raton, 2014. Available: https://www.routledge.com/Linear-Algebra-and-Matrix-Analysis-for-Statistics/Banerjee-Roy/p/book/9781420095388
[7]
Wikipedia contributors, “Bilinear form — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Bilinear_form
[8]
Wikipedia contributors, “Quadratic form — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Quadratic_form
[9]
Wikipedia contributors, “Trace (linear algebra) — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Trace_(linear_algebra)
[10]
Wikipedia contributors, “Frobenius norm — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Frobenius_norm
[11]
Wikipedia contributors, “Eigendecomposition of a matrix — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix
[12]
Wikipedia contributors, “Singular value decomposition — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Singular_value_decomposition
[13]
Wikipedia contributors, “Definite matrix — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Definite_matrix
[14]
Wikipedia contributors, “Condition number — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Condition_number
[15]
L. N. Trefethen and D. Bau III, Numerical linear algebra. SIAM, 1997. doi: 10.1137/1.9780898719574.
[16]
Wikipedia contributors, “Paraboloid — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Paraboloid
[17]
Wikipedia contributors, “Determinant — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Determinant