Linear and Affine Maps
1 Linear and affine maps
Definition 1 (Additive function) A function \(f: \mathbb{R}^p \to \mathbb{R}^m\) is additive [1] if it preserves vector addition: for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^p\),
\[f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) + f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})\]
Example 1 (An additive function, and one that is not)
\(f(x) = 2x\) on \(\mathbb{R}\) is additive. For \(x, y \in \mathbb{R}\):
\[ \begin{aligned} f(x + y) &= 2(x + y) \\ &= 2x + 2y \\ &= f(x) + f(y) \end{aligned} \]
\(g(x) = x + 1\) on \(\mathbb{R}\) is not additive, since \(g(0 + 0) = 1\) but \(g(0) + g(0) = 2\).
Exercise 1 (Testing whether \(x_1 x_2\) is additive) An additive function can be evaluated at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\) by evaluating it at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\) separately, so additivity is worth checking before splitting a computation that way. Let \(f: \mathbb{R}^2 \to \mathbb{R}\) be \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = x_1 x_2\).
Is \(f\) additive?
Try \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= (1, 0)\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}= (0, 1)\) in Definition 1.
Definition 2 (Homogeneous function) A function \(f: \mathbb{R}^p \to \mathbb{R}^m\) is homogeneous [1] if it preserves scalar multiplication: for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\) and \(c \in \mathbb{R}\),
\[f(c\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = c f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})\]
Example 2 (Homogeneous functions, and one that is not)
\(f(x) = 2x\) on \(\mathbb{R}\) is homogeneous. For \(x, c \in \mathbb{R}\):
\[ \begin{aligned} f(c x) &= 2 c x \\ &= c (2x) \\ &= c f(x) \end{aligned} \]
\(h(x) = x^2\) on \(\mathbb{R}\) is not homogeneous, since \(h(2 \cdot 1) = 4\) but \(2\,h(1) = 2\).
\(k(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathopen{}\left(x_1^3 + x_2^3\right)\mathclose{}^{1/3}\) on \(\mathbb{R}^2\) is homogeneous but not additive. For \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^2\) and \(c \in \mathbb{R}\):
\[ \begin{aligned} k(c \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= \mathopen{}\left(c^3 x_1^3 + c^3 x_2^3\right)\mathclose{}^{1/3} \\ &= \mathopen{}\left(c^3 \mathopen{}\left(x_1^3 + x_2^3\right)\mathclose{}\right)\mathclose{}^{1/3} \\ &= c \mathopen{}\left(x_1^3 + x_2^3\right)\mathclose{}^{1/3} \\ &= c\, k(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) \end{aligned} \]
but \(k((1, 0) + (0, 1)) = k((1, 1)) = 2^{1/3}\), while \(k((1, 0)) + k((0, 1)) = 2\).
Exercise 2 (Testing whether the absolute value is homogeneous) A homogeneous function turns a rescaled input into the same rescaling of its output, including a rescaling by a negative number. Let \(f: \mathbb{R}\to \mathbb{R}\) be the absolute value, \(f(x) = \mathopen{}\left|x\right|\mathclose{}\).
Is \(f\) homogeneous?
Try a negative \(c\) in Definition 2.
Definition 3 (Linear map, linear transformation, linear function) A function \(f: \mathbb{R}^p \to \mathbb{R}^m\) is a linear map [1] (also called a linear transformation or linear function) if it is both additive and homogeneous.
Example 3 (A linear map, and two functions that are not)
\(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (2 x_1,\ x_1 + x_2)\) on \(\mathbb{R}^2\) is linear. For \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^2\) and \(c \in \mathbb{R}\):
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) &= (2 (x_1 + y_1),\ (x_1 + y_1) + (x_2 + y_2)) \\ &= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) + f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) \end{aligned} \]
and
\[ \begin{aligned} f(c \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= (2 c x_1,\ c x_1 + c x_2) \\ &= c f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}). \end{aligned} \]
\(g(x) = x + 1\) on \(\mathbb{R}\) is not linear: additivity fails, since \(g(0 + 0) = 1\) but \(g(0) + g(0) = 2\).
\(h(x) = x^2\) on \(\mathbb{R}\) is not linear: \(h(2 \cdot 1) = 4\) but \(2\,h(1) = 2\), so \(h(c x) \ne c\,h(x)\) for \(c = 2\), \(x = 1\).
Exercise 3 (Testing whether a difference of scaled measurements is a linear map) A score that scales two measurements and takes their difference, such as \(3 x_1 - x_2\), is a common building block of a model. Let \(f: \mathbb{R}^2 \to \mathbb{R}\) be \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 3 x_1 - x_2\).
Is \(f\) a linear map?
Check the two conditions in Definition 3 one at a time, for any \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}\in \mathbb{R}^2\) and \(c \in \mathbb{R}\).
Definition 4 (Linear operator) A linear operator on \(\mathbb{R}^p\) is a linear map (Definition 3) from \(\mathbb{R}^p\) to itself, that is, a linear map \(f : \mathbb{R}^p \to \mathbb{R}^p\) whose input and output have the same length. Some authors use “operator” more loosely, for any linear map.
Example 4 (A linear operator, and a linear map that is not one) The map \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (2 x_1,\ x_1 + x_2)\) from Example 3 sends \(\mathbb{R}^2\) to \(\mathbb{R}^2\), so it is a linear operator on \(\mathbb{R}^2\). The linear map \(g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = x_1 + x_2\) sends \(\mathbb{R}^2\) to \(\mathbb{R}\), so it is linear but not an operator: its outputs have length \(1\), not \(2\).
Exercise 4 (Testing whether a projection is a linear operator) Dropping the last coordinate of a point in \(\mathbb{R}^3\) projects it onto a plane, as a shadow does. Let \(f: \mathbb{R}^3 \to \mathbb{R}^2\) be \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (x_1, x_2)\). This \(f\) is a linear map.
Is \(f\) a linear operator on \(\mathbb{R}^3\)?
Write \(f\) as \(f: \mathbb{R}^p \to \mathbb{R}^m\), and compare \(p\) with \(m\) as in Definition 4.
Theorem 1 (A linear map sends zero to zero) If \(f: \mathbb{R}^p \to \mathbb{R}^m\) is a linear map (Definition 3), then \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Proof. On the left, \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) is the zero vector of length \(p\); on the right, it is the zero vector of length \(m\).
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) &= f(0 \cdot\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) && \text{(every entry of } 0 \cdot\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\text{ is } 0 \cdot 0 = 0 \text{; }\href{linear-algebra-matrices.qmd#def-scalar-mult}{\text{Definition~9 in Matrices}}\text{)} \\&= 0 \cdot f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) && \text{(homogeneity, with } c = 0 \text{ and } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\text{)} \\&= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}} && \text{(every entry of } 0 \cdot f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) \text{ is } 0 \text{; }\href{linear-algebra-matrices.qmd#def-scalar-mult}{\text{Definition~9 in Matrices}}\text{)} \end{aligned} \]
Example 5 (Ruling out linearity at zero) In Example 3, \(g(x) = x + 1\) has \(g(0) = 1 \neq 0\), so by Theorem 1 it is not linear, without checking additivity or homogeneity.
Sending \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) to \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) is necessary but not sufficient: \(h(x) = x^2\) in the same example has \(h(0) = 0\) and is still not linear.
Exercise 5 (Ruling out a linear map from its value at zero) Theorem 1 gives a quick test: a function that does not send \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) to \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) cannot be a linear map. Let \(g: \mathbb{R}^2 \to \mathbb{R}^2\) be
\[ g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (x_1 + x_2,\ x_2 - 1). \]
Is \(g\) a linear map?
Compute \(g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}})\).
Theorem 2 (Multiplying by a matrix is a linear map) For any \(m \times p\) matrix \(\mathbf{A}\), the function \(f: \mathbb{R}^p \to \mathbb{R}^m\) defined by \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) is a linear map (Definition 3).
Proof. Both conditions are checked one entry at a time. For each \(i \in \{1, \ldots, m\}\), additivity:
\[ \begin{aligned} \mathopen{}\left(\mathbf{A}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})\right)\mathclose{}_i &= \sum_{j=1}^pa_{ij} (\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})_j && \text{(}\href{linear-algebra-matrix-multiplication.qmd#def-matvec-mult}{\text{Definition~8 in Matrix Multiplication}}\text{)} \\&= \sum_{j=1}^pa_{ij} (x_j + y_j) && \text{(}\href{linear-algebra-vectors.qmd#def-vector-addition}{\text{Definition~9 in Vectors}}\text{)} \\&= \sum_{j=1}^p(a_{ij} x_j + a_{ij} y_j) && \text{(multiplication distributes over addition)} \\&= \sum_{j=1}^pa_{ij} x_j + \sum_{j=1}^pa_{ij} y_j && \text{(a sum of sums splits)} \\&= (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i + (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})_i && \text{(}\href{linear-algebra-matrix-multiplication.qmd#def-matvec-mult}{\text{Definition~8 in Matrix Multiplication}}\text{)} \\&= (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})_i && \text{(}\href{linear-algebra-vectors.qmd#def-vector-addition}{\text{Definition~9 in Vectors}}\text{)} \end{aligned} \]
and homogeneity:
\[ \begin{aligned} \mathopen{}\left(\mathbf{A}(c\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})\right)\mathclose{}_i &= \sum_{j=1}^pa_{ij} (c\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_j && \text{(}\href{linear-algebra-matrix-multiplication.qmd#def-matvec-mult}{\text{Definition~8 in Matrix Multiplication}}\text{)} \\&= \sum_{j=1}^pa_{ij}\, c\, x_j && \text{(}\href{linear-algebra-matrices.qmd#def-scalar-mult}{\text{Definition~9 in Matrices}}\text{)} \\&= \sum_{j=1}^pc\, a_{ij} x_j && \text{(products are symmetric)} \\&= c \sum_{j=1}^pa_{ij} x_j && \text{(a constant factor comes out of a sum)} \\&= c\, (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i && \text{(}\href{linear-algebra-matrix-multiplication.qmd#def-matvec-mult}{\text{Definition~8 in Matrix Multiplication}}\text{)} \\&= (c\, \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i && \text{(}\href{linear-algebra-matrices.qmd#def-scalar-mult}{\text{Definition~9 in Matrices}}\text{)} \end{aligned} \]
The four algebra rules cited here are on the algebra page: distributivity, a sum of sums, symmetry of products, and a constant factor.
Exercise 6 (Recognizing a linear map as multiplication by a matrix) Theorem 2 saves checking additivity and homogeneity by hand: a function that multiplies its input by a fixed matrix is a linear map. Let \(f: \mathbb{R}^2 \to \mathbb{R}^2\) be
\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (x_1 + 2 x_2,\ 3 x_1 - x_2). \]
Is \(f\) a linear map?
Find a \(2 \times 2\) matrix \(\mathbf{A}\) with \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\), using Definition 8 in Matrix Multiplication.
Theorem 3 (A linear map preserves linear combinations) If \(f: \mathbb{R}^p \to \mathbb{R}^m\) is a linear map (Definition 3), then for any vectors \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1, \ldots, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_k \in \mathbb{R}^p\) and any coefficients \(c_1, \ldots, c_k \in \mathbb{R}\):
\[f\mathopen{}\left(\sum_{j=1}^{k} c_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j\right)\mathclose{} = \sum_{j=1}^{k} c_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j)\]
Proof. By induction on \(k\). For \(k = 1\), the claim \(f(c_1 \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1) = c_1 f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1)\) is homogeneity.
Suppose the claim holds for \(k\) vectors. Then for \(k + 1\):
\[ \begin{aligned} f\mathopen{}\left(\sum_{j=1}^{k+1} c_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j\right)\mathclose{} &= f\mathopen{}\left(\sum_{j=1}^{k} c_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j + c_{k+1} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_{k+1}\right)\mathclose{} && \text{(split off the last term)} \\&= f\mathopen{}\left(\sum_{j=1}^{k} c_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j\right)\mathclose{} + f(c_{k+1} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_{k+1}) && \text{(additivity)} \\&= \sum_{j=1}^{k} c_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j) + f(c_{k+1} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_{k+1}) && \text{(induction hypothesis)} \\&= \sum_{j=1}^{k} c_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j) + c_{k+1} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_{k+1}) && \text{(homogeneity)} \\&= \sum_{j=1}^{k+1} c_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_j) && \text{(join the last term back on)} \end{aligned} \]
Exercise 7 (Evaluating a linear map on a linear combination) A linear map is determined on a linear combination of vectors by its values on those vectors alone. Let \(f: \mathbb{R}^2 \to \mathbb{R}^2\) be a linear map with
\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1) = \begin{bmatrix} 1 \\ 2 \end{bmatrix}, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_2) = \begin{bmatrix} 0 \\ -1 \end{bmatrix}. \]
Find \(f(3 \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_1 - 2 \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}}_2)\).
Apply Theorem 3 with \(c_1 = 3\) and \(c_2 = -2\).
Theorem 4 (Every linear map is multiplication by a matrix) If \(f: \mathbb{R}^p \to \mathbb{R}^m\) is a linear map (Definition 3), then there is exactly one \(m \times p\) matrix \(\mathbf{A}\) such that
\[f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\quad \text{for all } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\]
Column \(j\) of \(\mathbf{A}\) is \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\), the value of \(f\) at the \(j\)th indicator vector (Definition 21 in Vectors); that is, \(a_{ij} = \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i\).
Proof. Existence. Let \(\mathbf{A}\) be the \(m \times p\) matrix with \(a_{ij} = \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i\). For any \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\),
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= f\mathopen{}\left(\sum_{j=1}^px_j \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\right)\mathclose{} && \text{(}\href{linear-algebra-vectors.qmd#thm-standard-basis-expansion}{\text{Theorem~4 in Vectors}}\text{)} \\&= \sum_{j=1}^px_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j) && \text{(}\href{#thm-linear-map-lincom}{\text{Theorem~3}}\text{)} \end{aligned} \]
and entry \(i\) of that sum is
\[ \begin{aligned} \mathopen{}\left(\sum_{j=1}^px_j f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i &= \sum_{j=1}^px_j \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i && \text{(}\href{linear-algebra-vectors.qmd#def-linear-combination}{\text{Definition~11 in Vectors}}\text{, entry by entry)} \\&= \sum_{j=1}^px_j a_{ij} && \text{(definition of } \mathbf{A} \text{)} \\&= \sum_{j=1}^pa_{ij} x_j && \text{(products are symmetric)} \\&= (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})_i && \text{(}\href{linear-algebra-matrix-multiplication.qmd#def-matvec-mult}{\text{Definition~8 in Matrix Multiplication}}\text{)} \end{aligned} \]
Uniqueness. Suppose \(\mathbf{B}\) is any \(m \times p\) matrix with \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\). Taking \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j\), for each \(i\):
\[ \begin{aligned} \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i &= (\mathbf{B}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)_i && \text{(assumption on } \mathbf{B} \text{)} \\&= \sum_{k=1}^{p} b_{ik} (\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)_k && \text{(}\href{linear-algebra-matrix-multiplication.qmd#def-matvec-mult}{\text{Definition~8 in Matrix Multiplication}}\text{)} \\&= b_{ij} \cdot 1 + \sum_{k \neq j} b_{ik} \cdot 0 && \text{(}\href{linear-algebra-vectors.qmd#def-indicator-vector}{\text{Definition~21 in Vectors}}\text{, splitting off the } k = j \text{ term)} \\&= b_{ij} + 0 && \text{(times } 1 \text{ changes nothing; times } 0 \text{ gives } 0 \text{)} \\&= b_{ij} && \text{(adding } 0 \text{ changes nothing)} \end{aligned} \]
So
\[ \begin{aligned} b_{ij} &= \mathopen{}\left(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\right)\mathclose{}_i \\ &= a_{ij} \end{aligned} \]
for every \(i\) and \(j\), and \(\mathbf{B} = \mathbf{A}\).
Definition 5 (Matrix of a linear map) The matrix of a linear map \(f: \mathbb{R}^p \to \mathbb{R}^m\) is the unique \(m \times p\) matrix \(\mathbf{A}\) with \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) (Theorem 4). Its column \(j\) is \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\).
Example 6 (Finding the matrix of a linear map) For \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (2 x_1,\ x_1 + x_2)\) from Example 3, the values of \(f\) at the indicator vectors are
\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_1) = f\mathopen{}\left(\begin{bmatrix}1 \\ 0\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}2 \cdot 1 \\ 1 + 0\end{bmatrix} = \begin{bmatrix}2 \\ 1\end{bmatrix}, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_2) = f\mathopen{}\left(\begin{bmatrix}0 \\ 1\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}2 \cdot 0 \\ 0 + 1\end{bmatrix} = \begin{bmatrix}0 \\ 1\end{bmatrix} \]
and these are the columns of the matrix of \(f\) (Definition 5): \(\mathbf{A} = \begin{bmatrix}2 & 0 \\ 1 & 1\end{bmatrix}\).
Exercise 8 (Evaluating a linear map from its values on the indicator vectors) By Theorem 4, the values of a linear map at the indicator vectors determine it everywhere. Let \(f: \mathbb{R}^2 \to \mathbb{R}^2\) be a linear map with
\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_1) = \begin{bmatrix} 1 \\ 3 \end{bmatrix}, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_2) = \begin{bmatrix} -2 \\ 0 \end{bmatrix}. \]
Find \(f\mathopen{}\left(\begin{bmatrix} 4 \\ 1 \end{bmatrix}\right)\mathclose{}\).
Build the matrix of \(f\) from the two given values, then multiply.
Exercise 9 (Finding the matrix of a linear map into \(\mathbb{R}^3\)) The matrix of a linear map turns the map into a table of numbers that software can store and multiply. Let \(f: \mathbb{R}^2 \to \mathbb{R}^3\) be the linear map
\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (x_2,\ x_1,\ x_1 - x_2). \]
Find the matrix of \(f\).
Column \(j\) of the matrix is \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\).
Corollary 1 (Linear functions from vectors to numbers) A function \(f: \mathbb{R}^p \to \mathbb{R}\) is linear (Definition 3) if and only if there is a vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \in \mathbb{R}^p\) such that
\[f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} \quad \text{for all } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\]
and then \(w_j = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\) for each \(j\).
Proof. If \(f\) is linear. Theorem 4 with \(m = 1\) gives a \(1 \times p\) matrix \(\mathbf{A}\) with \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and \(a_{1j} = f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j)\). Let \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\) be the vector with \(w_j = a_{1j}\), so that \(\mathbf{A} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top}\). Then
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && \text{(}\href{#thm-linear-map-matrix}{\text{Theorem~4}}\text{)} \\&= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} && (\mathbf{A} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top}) \end{aligned} \]
and
\[ \begin{aligned} w_j &= a_{1j} \\ &= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_j). \end{aligned} \]
If \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\). Here \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top}\) is a \(1 \times p\) matrix, so \(f\) is linear by Theorem 2.
Example 7 (A linear function from \(\mathbb{R}^2\) to \(\mathbb{R}\)) \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 3 x_1 + 5 x_2\) is \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) with \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} = {(3, 5)}^{\top}\), so it is linear by Corollary 1. The entries of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\) are the values at the indicator vectors:
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_1) &= 3 \cdot 1 + 5 \cdot 0 \\ &= 3 \end{aligned} \]
and
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_2) &= 3 \cdot 0 + 5 \cdot 1 \\ &= 5. \end{aligned} \]
\(g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 3 x_1 + 5 x_2 + 1\) is not linear, because \(g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = 1 \neq 0\) (Theorem 1).
Exercise 10 (Finding the weight vector of a linear score) A linear score of \(p\) measurements is a dot product of the measurements with a vector of weights, by Corollary 1. Let \(f: \mathbb{R}^3 \to \mathbb{R}\) be a linear map with
\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_1) = 2, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_2) = 0, \qquad f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{e}}\vphantom{e}}_3) = -1. \]
Find the vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\) with \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) for every \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^3\).
Corollary 1 gives each entry \(w_j\) of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\).
Two chapters of the Essence of linear algebra series by 3Blue1Brown show the results above geometrically:
- Linear transformations and matrices (Chapter 3) shows why a linear map is fixed by where it sends the indicator vectors, which become the columns of its matrix (Definition 5).
- Dot products and duality (Chapter 9) shows why every linear map from \(\mathbb{R}^p\) to \(\mathbb{R}\) is a dot product with a fixed vector (Corollary 1).
Definition 6 (Translation) For a fixed vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \in \mathbb{R}^m\), the translation [2] by \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) is the function \(t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}} : \mathbb{R}^m \to \mathbb{R}^m\) defined by
\[t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\]
Example 8 (Translating two points) The translation by \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = {(1, -2)}^{\top}\) sends
\[ t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}\mathopen{}\left(\begin{bmatrix}0 \\ 0\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}0 + 1 \\ 0 + (-2)\end{bmatrix} = \begin{bmatrix}1 \\ -2\end{bmatrix}, \qquad t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}\mathopen{}\left(\begin{bmatrix}3 \\ 4\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}3 + 1 \\ 4 + (-2)\end{bmatrix} = \begin{bmatrix}4 \\ 2\end{bmatrix} \]
Both points move one unit right and two units down.
Remark 1 (A translation moves every point by the same vector). A translation by \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) moves every point by the same vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\).
Exercise 11 (Finding the translation that moves one point to another) A translation moves every point by the same vector, so where it sends one point fixes the whole translation. A translation \(t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}\) of \(\mathbb{R}^2\) sends \({(2, 3)}^{\top}\) to \({(5, 1)}^{\top}\).
Find \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\).
Solve \({(2, 3)}^{\top} + \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = {(5, 1)}^{\top}\) entry by entry.
Definition 7
Affine map, affine transformation, affine function
A function \(f: \mathbb{R}^p \to \mathbb{R}^m\) is an affine map (also called an affine transformation or affine function) if it is a matrix product followed by a translation (Definition 6) (a composition, with the matrix product as the inner function): there are an \(m \times p\) matrix \(\mathbf{A}\) and a vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \in \mathbb{R}^m\) such that
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= t_{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}}(\mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) \\ &= \mathbf{A} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \quad \text{for all } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p \end{aligned} \]
Exercise 12 (Testing whether a rescale-and-shift is an affine map) Rescaling a measurement and then shifting it, as when converting units with a different zero point, is the typical affine map. Let \(f: \mathbb{R}^2 \to \mathbb{R}^2\) be
\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (2 x_1 + 1,\ x_2 - 3). \]
Is \(f\) an affine map?
Look for a \(2 \times 2\) matrix \(\mathbf{A}\) and a vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) with \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\).
Definition 8 (Offset of an affine map, intercept, bias) In Definition 7, the vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) is called the offset of \(f\) (statistics also calls it the intercept, and machine learning the bias).
Remark 2 (Real-valued affine maps). For a real-valued function (Definition 6 in Vectors, \(m = 1\)), \(\mathbf{A}\) is a single row, written \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top}\), and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) is a single number \(b\), so an affine function is a linear function \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) (Corollary 1) plus an offset:
\[f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b, \qquad \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \in \mathbb{R}^p, \quad b \in \mathbb{R} \tag{1}\]
Exercise 13 (Reading off the offset of an affine map) In a fitted model, the offset is the part of the output that does not depend on the input. Let \(f: \mathbb{R}^2 \to \mathbb{R}^2\) be the affine map
\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (x_1 - x_2 + 4,\ 3 x_2 - 1). \]
Find the offset of \(f\).
Write \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})\) as \(\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) as in Definition 7.
Definition 9 (Augmented vector) The augmented vector of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\) is the vector \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} \in \mathbb{R}^{p+1}\) formed by placing a \(1\) above the entries of \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\):
\[ \begin{aligned} {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} &= \begin{bmatrix}1 \\ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\end{bmatrix} \\ &= {(1, x_1, \ldots, x_p)}^{\top} \end{aligned} \]
Example 9 (An affine function as a dot product with an augmented vector) For \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}= {(2, 5)}^{\top}\), the augmented vector is \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} = {(1, 2, 5)}^{\top}\). With \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} = {(3, -1)}^{\top}\) and \(b = 4\), the affine function \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b\) (Equation 1) equals the dot product of \({(b, w_1, w_2)}^{\top} = {(4, 3, -1)}^{\top}\) with \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}}\):
\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b &= 3 \cdot 2 + (-1) \cdot 5 + 4 \\ &= 5, \\ {(4, 3, -1)}^{\top} \cdot {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}} &= 4 \cdot 1 + 3 \cdot 2 + (-1) \cdot 5 \\ &= 5 \end{aligned} \]
Exercise 14 (Absorbing an offset into a dot product with the augmented vector) The augmented vector lets software treat an affine score as a dot product, with the offset as one more weight. Let \(f: \mathbb{R}^2 \to \mathbb{R}\) be
\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = 2 x_1 - x_2 + 5. \]
Find the vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \in \mathbb{R}^3\) with \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{v}}\vphantom{v}} \cdot {\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}}\) for every \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^2\).
Follow Example 9. The first entry of \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}}\) is \(1\).
Theorem 5 (An affine map sends zero to its offset) An affine map \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) (Definition 7) satisfies \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\).
Proof. First, \(\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\): for each \(i\),
\[ \begin{aligned} (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}})_i &= \sum_{j=1}^pa_{ij} \cdot 0 && \text{(}\href{linear-algebra-matrix-multiplication.qmd#def-matvec-mult}{\text{Definition~8 in Matrix Multiplication}}\text{)} \\&= \sum_{j=1}^p0 && \text{(any number times } 0 \text{ is } 0 \text{)} \\&= 0 && \text{(a sum of zeros is } 0 \text{)} \end{aligned} \]
Then
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} && \text{(}\href{#def-affine-map}{\text{Definition~7}}\text{)} \\&= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} && (\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) \\&= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} && \text{(adding } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\text{ changes no entry; }\href{linear-algebra-vectors.qmd#def-vector-addition}{\text{Definition~9 in Vectors}}\text{)} \end{aligned} \]
Exercise 15 (Recovering a column of an affine map’s matrix from two values) Theorem 5 says that the value of an affine map at \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) is its offset, so subtracting that value isolates the matrix part. Let \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) be an affine map from \(\mathbb{R}^2\) to \(\mathbb{R}^2\) with
\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \begin{bmatrix}1 \\ 2\end{bmatrix}, \qquad f\mathopen{}\left(\begin{bmatrix}1 \\ 0\end{bmatrix}\right)\mathclose{} = \begin{bmatrix}4 \\ 2\end{bmatrix}. \]
Find the first column of \(\mathbf{A}\).
\(\mathbf{A} \begin{bmatrix}1 \\ 0\end{bmatrix}\) is the first column of \(\mathbf{A}\).
Theorem 6 (An affine map is linear exactly when its offset is zero) An affine map \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) (Definition 7) is linear (Definition 3) if and only if \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Proof. If \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), then
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\\ &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, \end{aligned} \]
which is linear by Theorem 2. Conversely, if \(f\) is linear, then \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) by Theorem 1, and \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) by Theorem 5, so \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Example 10 (Two affine maps with the same matrix) Let \(\mathbf{A} = \begin{bmatrix}2 & 0 \\ 1 & 1\end{bmatrix}\), the matrix of the linear map in Example 3. \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) has offset \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), so it is linear. \(g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \begin{bmatrix}1 \\ 0\end{bmatrix}\) has offset \(\begin{bmatrix}1 \\ 0\end{bmatrix} \neq \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), so by Theorem 6 it is not linear: indeed \(g(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) = \begin{bmatrix}1 \\ 0\end{bmatrix}\), not \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Exercise 16 (Choosing a constant that makes an affine map linear) By Theorem 6, an affine map is a linear map exactly when its offset is \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\). For \(c \in \mathbb{R}\), let \(f: \mathbb{R}^2 \to \mathbb{R}^2\) be
\[ f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = (x_1 - x_2 + c - 2,\ 3 x_1 + c^2 - 4). \]
For which values of \(c\) is \(f\) linear?
Find the offset of \(f\) as a function of \(c\).
Corollary 2 (Every linear map is affine) Every linear map \(f : \mathbb{R}^p \to \mathbb{R}^m\) (Definition 3) is an affine map (Definition 7) with offset \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Proof. If \(f\) is linear, Theorem 4 gives a matrix \(\mathbf{A}\) with
\[ \begin{aligned} f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\\ &= \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}, \end{aligned} \]
which is Definition 7 with \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Exercise 17 (Testing whether a linear score is an affine map) Some results hold only for affine maps, so it helps to check whether a given score qualifies. Let \(f: \mathbb{R}^2 \to \mathbb{R}\) be \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = x_1 - 4 x_2\), a linear map.
Is \(f\) an affine map?
Use Corollary 2.
Theorem 7 (An affine map determines its matrix and offset) If \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) and \(f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) = \mathbf{A}'\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}'\) for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\) (Definition 7), then \(\mathbf{A}' = \mathbf{A}\) and \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}' = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\).
Proof. By Theorem 5, applied to each formula,
\[ \begin{aligned} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} &= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}) \\ &= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}'. \end{aligned} \]
Then
\[ \begin{aligned} \mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}&= f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) - \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}} \\ &= \mathbf{A}'\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} \end{aligned} \]
for all \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\), so \(\mathbf{A}\) and \(\mathbf{A}'\) are both matrices (Definition 5) of the linear map \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\mapsto f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) - \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\) (linear by Theorem 2), and the uniqueness in Theorem 4 gives \(\mathbf{A}' = \mathbf{A}\).
Exercise 18 (Matching the offsets of two formulas for one affine map) Theorem 7 says that an affine map has only one matrix and one offset, so two ways of writing the same map must agree term by term. For \(c \in \mathbb{R}\), suppose that
\[ \begin{bmatrix}1 & 0 \\ 2 & 1\end{bmatrix} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \begin{bmatrix}3 \\ 1\end{bmatrix} = \begin{bmatrix}1 & 0 \\ 2 & 1\end{bmatrix} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \begin{bmatrix}c \\ 1\end{bmatrix} \quad \text{for every } \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^2. \]
Find \(c\).
Compare the offsets of the two sides.
Remark 3 (“Linear” in this site, and in machine learning and statistics). This site uses “linear” only in the sense of Definition 3, so by Theorem 6 a function with a nonzero offset is affine but not linear. For functions of one variable, this is the same split as affine \(a x + b\) versus linear \(a x\) on the algebra page. (The algebra page writes these as \(m x + b\) and \(m x\); here \(m\) already names the number of outputs.)
Machine learning and statistics often call any function of the form Equation 1 “linear”, as in a linear model in statistics or a linear layer in a neural network, and call the offset \(b\) the bias (machine learning) or the intercept (statistics), as Definition 7 notes. A model with an intercept is affine in \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) but linear in its coefficients \((b, \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}})\), and that is the sense of “linear” in “linear model”: it stays linear in the coefficients even when the inputs are transformed: \(w_1 x + w_2 x^2 + b\) is not affine in \(x\), but it is linear in \((b, w_1, w_2)\). Separately, the affine function \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b\) is a linear function of the augmented vector \({\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}}^{\mathrm{aug}}\) (Definition 9), with \(b\) as the coefficient on its first entry.
- Zhou [3], sections 6 (“Linear functions and operators”; see Definition 4) and 7 (“Affine functions”), proves that every linear function is a matrix-vector product by the same route as Theorem 4, and that every affine function has the form \(\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\). It defines “affine” differently, by \(f(\alpha\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \beta\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}}) = \alpha f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) + \beta f(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{y}}\vphantom{y}})\) whenever \(\alpha+ \beta= 1\), and shows that this condition holds exactly for the maps \(\mathbf{A}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}+ \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{b}}\vphantom{b}}\), so the two definitions agree.
- Boyd and Vandenberghe [4], chapter 2 (“Linear functions”), covers the vector-to-number case: linear functions as inner products, affine functions, the first-order Taylor approximation as an affine function, and the regression model. Section 8.1 (“Linear and affine functions”) covers the vector-to-vector case. A free PDF is on the book’s website.
Definition 10 (Hyperplane) A hyperplane [5] in \(\mathbb{R}^p\) is the set of points \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^p\) satisfying
\[\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b = 0\]
for some vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \in \mathbb{R}^p \setminus \{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\}\) and number \(b \in \mathbb{R}\). The number \(b\) is the offset of the hyperplane.
Definition 11 (Normal vector of a hyperplane) In Definition 10, the vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\) is a normal vector [6] of the hyperplane.
Example 11 (A line in \(\mathbb{R}^2\), and why the normal vector must be nonzero)
- With \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} = (1, 1)\) and \(b = -1\), the hyperplane in \(\mathbb{R}^2\) is the line \(x_1 + x_2 - 1 = 0\). The point \((1, 0)\) is on it, since \(1 + 0 - 1 = 0\); the point \((1, 1)\) is not, since \(1 + 1 - 1 = 1 > 0\), so it lies in the half-space where \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \cdot \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}} + b > 0\).
- With \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} = \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\), the equation reads \(b = 0\): if \(b \ne 0\) no point satisfies it, and if \(b = 0\) every point does, so the set is empty or all of \(\mathbb{R}^2\), not a line. That degenerate case is why the definition requires \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \ne \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\).
Exercise 19 (Testing whether a vertical line is a hyperplane) A vertical line in \(\mathbb{R}^2\) is parallel to the \(x_2\)-axis. It cannot be written as \(x_2 = m x_1 + k\) for any numbers \(m\) and \(k\). Let \(L\) be the set of points \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^2\) with \(x_1 = 3\).
Is \(L\) a hyperplane in \(\mathbb{R}^2\)?
Look for \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \ne \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{0}}\vphantom{0}}\) and \(b\) in Definition 10.
Exercise 20 (Finding a normal vector of a plane in \(\mathbb{R}^3\)) In \(\mathbb{R}^3\) a hyperplane is a plane, and a normal vector points straight out of it. Let \(H\) be the hyperplane of points \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\in \mathbb{R}^3\) with
\[ 2 x_1 - x_2 + 4 x_3 = 7. \]
Find a normal vector of \(H\).
Rewrite the equation in the form of Definition 10.
Exercise 21 (Write a rule in matrix form) A clinic scores a patient’s risk from three measurements.
| symbol | meaning | units |
|---|---|---|
| \(x_1\) | age | years |
| \(x_2\) | systolic blood pressure | mmHg |
| \(x_3\) | body mass index | kg/m\(^2\) |
The rule in use is
\[\text{score} = 0.4\,x_1 + 0.2\,x_2 + 1.5\,x_3 - 30\]
Write this rule in the form of Equation 1, naming \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\), \(b\) and \(p\) explicitly.
Exercise 22 (Score a patient) Give the score from the rule in Exercise 21 for a patient aged \(50\) with blood pressure \(130\) and BMI \(28\).
Exercise 23 (Which of these are affine in \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\)?) Each of the five expressions is a function of the weights \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}} \in \mathbb{R}^p\), with the data \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and \(y\) held fixed. Say which ones are affine (Definition 7), that is, have the form Equation 1 — an inner product plus an offset — and which of those are also linear (Definition 3).
- \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\)
- \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}- y\)
- \((\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}^{\top} \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}- y)^2\)
- \(\mathopen{}\left\lVert\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{w}}\vphantom{w}}\right\rVert\mathclose{}_2^2\)
- \(w_1 x_1 + 3\)
Definitions on this page
- Definition 1 (Additive function)
- Definition 2 (Homogeneous function)
- Definition 3 (Linear map, linear transformation, linear function)
- Definition 4 (Linear operator)
- Definition 5 (Matrix of a linear map)
- Definition 6 (Translation)
- Definition 7
- Definition 8 (Offset of an affine map, intercept, bias)
- Definition 9 (Augmented vector)
- Definition 10 (Hyperplane)
- Definition 11 (Normal vector of a hyperplane)