Last modified: 2026-09-28 23:45:45 (PDT)
(adapted from Fieller (2016), Section 7.2)
Let \(\tilde{x}\) and \(\tilde{\beta}\) be column vectors of length \(p\) (see column vector and dot product).
Definition 1 (Vector derivative) If \(f(\tilde{\beta})\) is a scalar-valued function of a \(p \times 1\) vector \(\tilde{\beta}\), such as \(f(\tilde{\beta}) = {\tilde{x}}^{\top}\tilde{\beta}\), then its vector derivative is:
\[ \frac{\partial}{\partial \tilde{\beta}} f(\tilde{\beta}) = \begin{bmatrix} \frac{\partial}{\partial \beta_1}f(\tilde{\beta}) \\ \frac{\partial}{\partial \beta_2}f(\tilde{\beta}) \\ \vdots \\ \frac{\partial}{\partial \beta_p}f(\tilde{\beta}) \end{bmatrix} \]
Video lecture
Hutchinson’s Gradients Refresher (17 min) covers gradients and partial derivatives, the ideas behind this section (Hutchinson, n.d.). The login for the video site is posted on Canvas.
Definition 2 (Row-vector derivative) If \(f(\tilde{\beta})\) is a scalar-valued function of a \(p \times 1\) vector \(\tilde{\beta}\), such as \(f(\tilde{\beta}) = {\tilde{x}}^{\top}\tilde{\beta}\), then its row-vector derivative is:
\[ \frac{\partial}{\partial \tilde{\beta}^{\top}} f(\tilde{\beta}) = \begin{bmatrix} \frac{\partial}{\partial \beta_1}f(\tilde{\beta}) & \frac{\partial}{\partial \beta_2}f(\tilde{\beta}) & \cdots & \frac{\partial}{\partial \beta_p}f(\tilde{\beta}) \end{bmatrix} \]
Theorem 1 (Row and column derivatives are transposes) \[\frac{\partial}{\partial \tilde{\beta}^{\top}} f(\tilde{\beta}) = \mathopen{}\left(\frac{\partial}{\partial \tilde{\beta}} f(\tilde{\beta})\right)\mathclose{}^{\top}\]
\[\frac{\partial}{\partial \tilde{\beta}} f(\tilde{\beta}) = \mathopen{}\left(\frac{\partial}{\partial \tilde{\beta}^{\top}} f(\tilde{\beta})\right)\mathclose{}^{\top}\]
Proof. By Definition 1 and Definition 2, entry \(j\) of both \(\frac{\partial}{\partial \tilde{\beta}} f(\tilde{\beta})\) and \(\frac{\partial}{\partial \tilde{\beta}^{\top}} f(\tilde{\beta})\) is \(\frac{\partial}{\partial \beta_j} f(\tilde{\beta})\); the first is a \(p \times 1\) column and the second a \(1 \times p\) row with the same entries in the same order, so each is the transpose of the other.
Example 1 (Row and column derivatives of a linear function) For \(f(\tilde{\beta}) = 3\beta_1 + 5\beta_2\):
\[ \frac{\partial}{\partial \tilde{\beta}} f(\tilde{\beta}) = \begin{bmatrix}3 \\ 5\end{bmatrix}, \qquad \frac{\partial}{\partial \tilde{\beta}^{\top}} f(\tilde{\beta}) = \begin{bmatrix}3 & 5\end{bmatrix}, \]
and each is the transpose of the other.
Definition 3 (Derivative of a vector-valued function) If \(\tilde{y}= \tilde{y}(\tilde{\beta}) = {(y_1, \ldots, y_q)}^{\top}\) is a \(q \times 1\) vector-valued function of the \(p \times 1\) vector \(\tilde{\beta}\), its derivative with respect to \(\tilde{\beta}\) is the \(p \times q\) matrix whose \((i, j)\) entry is
\[ \mathopen{}\left[\frac{\partial}{\partial \tilde{\beta}} {\tilde{y}}^{\top}\right]\mathclose{}_{ij} \stackrel{\text{def}}{=}\frac{\partial}{\partial \beta_i} y_j, \qquad i = 1, \ldots, p, \quad j = 1, \ldots, q. \]
These notes use this denominator layout throughout: rows index the entries of \(\tilde{\beta}\) (the denominator) and columns index the entries of \(\tilde{y}\) (the numerator), so column \(j\) is the vector derivative \(\frac{\partial}{\partial \tilde{\beta}} y_j\) (Definition 1). Both \(\frac{\partial}{\partial \tilde{\beta}} \tilde{y}\) and \(\frac{\partial}{\partial \tilde{\beta}} {\tilde{y}}^{\top}\) denote this \(p \times q\) matrix.
Example 2 (Differentiating a \(3 \times 1\) function of a \(2 \times 1\) vector) Let \(\tilde{\beta}= {(\beta_1, \beta_2)}^{\top}\) (\(p = 2\)) and \(\tilde{y}(\tilde{\beta}) = {(\beta_1^2,\ \beta_1\beta_2,\ 3\beta_2)}^{\top}\) (\(q = 3\)). Then
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}} {\tilde{y}}^{\top}}_{2 \times 3} = \begin{bmatrix} \frac{\partial}{\partial \beta_1} \beta_1^2 & \frac{\partial}{\partial \beta_1} \beta_1\beta_2 & \frac{\partial}{\partial \beta_1} 3\beta_2 \\ \frac{\partial}{\partial \beta_2} \beta_1^2 & \frac{\partial}{\partial \beta_2} \beta_1\beta_2 & \frac{\partial}{\partial \beta_2} 3\beta_2 \end{bmatrix} = \begin{bmatrix} 2\beta_1 & \beta_2 & 0 \\ 0 & \beta_1 & 3 \end{bmatrix} \]
Definition 4 (Constant) A \(q \times 1\) vector \(\tilde{x}\) is constant with respect to the \(p \times 1\) vector \(\tilde{\beta}\) if its derivative (Definition 3) is zero:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}} {\tilde{x}}^{\top}}_{p \times q} = \underbrace{\mathbf{0}}_{p \times q} \]
Example 3 (A constant vector) Let \(\tilde{\beta}= {(\beta_1, \beta_2)}^{\top}\) and \(\tilde{x}= {(3, 5)}^{\top}\), so \(x_1 = 3\) and \(x_2 = 5\) do not depend on \(\tilde{\beta}\). Expanding \(\frac{\partial}{\partial \tilde{\beta}} {\tilde{x}}^{\top}\) into its matrix of scalar partial derivatives (Definition 3) and evaluating each entry:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}} {\tilde{x}}^{\top}}_{2 \times 2} = \frac{\partial}{\partial \tilde{\beta}} \begin{bmatrix}x_1 & x_2\end{bmatrix} = \begin{bmatrix} \frac{\partial}{\partial \beta_1} x_1 & \frac{\partial}{\partial \beta_1} x_2 \\ \frac{\partial}{\partial \beta_2} x_1 & \frac{\partial}{\partial \beta_2} x_2 \end{bmatrix} = \begin{bmatrix} \frac{\partial}{\partial \beta_1} 3 & \frac{\partial}{\partial \beta_1} 5 \\ \frac{\partial}{\partial \beta_2} 3 & \frac{\partial}{\partial \beta_2} 5 \end{bmatrix} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \end{bmatrix} = \underbrace{\mathbf{0}}_{2 \times 2} \]
Every entry is the derivative of a constant, so \(\frac{\partial}{\partial \tilde{\beta}} {\tilde{x}}^{\top} = \underbrace{\mathbf{0}}_{2 \times 2}\) and \(\tilde{x}\) is constant with respect to \(\tilde{\beta}\) (Definition 4).
Theorem 2 (Derivative of a dot product) If \(\tilde{x}\) is constant with respect to \(\tilde{\beta}\), then:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}} (\tilde{x}\cdot \tilde{\beta})}_{p \times 1} = \underbrace{\frac{\partial}{\partial \tilde{\beta}} (\tilde{\beta}\cdot \tilde{x})}_{p \times 1} = \underbrace{\tilde{x}}_{p \times 1} \]
Proof. \[ \begin{aligned} \frac{\partial}{\partial \tilde{\beta}} (\tilde{x}\cdot \tilde{\beta}) &= \begin{bmatrix} \frac{\partial}{\partial \beta_1}(x_1\beta_1+x_2\beta_2 +...+x_p \beta_p ) \\ \frac{\partial}{\partial \beta_2}(x_1\beta_1+x_2\beta_2 +...+x_p \beta_p ) \\ \vdots \\ \frac{\partial}{\partial \beta_p}(x_1\beta_1+x_2\beta_2 +...+x_p \beta_p ) \end{bmatrix} \\ &= \begin{bmatrix} x_{1} \\ x_{2} \\ \vdots \\ x_{p} \end{bmatrix} \\ &= \tilde{x} \end{aligned} \]
Example 4 (Derivative of a dot product) Let \(\tilde{x}= {(3, 5)}^{\top}\) (constant with respect to \(\tilde{\beta}\); see Example 3) and \(\tilde{\beta}= {(\beta_1, \beta_2)}^{\top}\). Then \(\tilde{x}\cdot \tilde{\beta}= 3\beta_1 + 5\beta_2\), and by Theorem 2:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}}(\tilde{x}\cdot \tilde{\beta})}_{2 \times 1} = \underbrace{\tilde{x}}_{2 \times 1} = \begin{pmatrix} 3 \\ 5 \end{pmatrix} \]
Verifying entry-wise:
\[ \frac{\partial}{\partial \tilde{\beta}}(3\beta_1 + 5\beta_2) = \begin{pmatrix} \frac{\partial}{\partial \beta_1}(3\beta_1 + 5\beta_2) \\ \frac{\partial}{\partial \beta_2}(3\beta_1 + 5\beta_2) \end{pmatrix} = \begin{pmatrix} 3 \\ 5 \end{pmatrix} \]
Both methods agree.
Theorem 3 (Product rule for dot-products) If \(\tilde{a} = \tilde{a}(\tilde{x})\) and \(\tilde{b} = \tilde{b}(\tilde{x})\) are differentiable \(p \times 1\) vector functions of \(\tilde{x}\), then:
\[ \begin{aligned} \frac{\partial}{\partial \underbrace{\tilde{x}}_{p \times 1}} \underbrace{\tilde{a}}_{p \times 1} \cdot \underbrace{\tilde{b}}_{p \times 1} &= \mathopen{}\left( \frac{\partial}{\partial \underbrace{\tilde{x}}_{p \times 1}} \underbrace{{\tilde{a}}^{\top}}_{1 \times p} \right)\mathclose{} \underbrace{\tilde{b}}_{p \times 1} + \mathopen{}\left( \frac{\partial}{\partial \underbrace{\tilde{x}}_{p \times 1}} \underbrace{{\tilde{b}}^{\top}}_{1 \times p} \right)\mathclose{} \underbrace{\tilde{a}}_{p \times 1} \end{aligned} \]
Proof. Entry-wise, for \(i = 1, \ldots, p\):
\[ \begin{aligned} \left[\frac{\partial}{\partial \tilde{x}} (\tilde{a} \cdot \tilde{b})\right]_i &= \frac{\partial}{\partial x_i} \sum_{k=1}^p a_k b_k \\ &= \sum_{k=1}^p \mathopen{}\left(b_k \frac{\partial}{\partial x_i} a_k + a_k \frac{\partial}{\partial x_i} b_k\right)\mathclose{} \\ &= \left[\mathopen{}\left(\frac{\partial}{\partial \tilde{x}} {\tilde{a}}^{\top}\right)\mathclose{}\tilde{b}\right]_i + \left[\mathopen{}\left(\frac{\partial}{\partial \tilde{x}} {\tilde{b}}^{\top}\right)\mathclose{}\tilde{a}\right]_i \end{aligned} \]
Example 5 (Example of the dot-product rule) Apply Theorem 3 with the vector \(\tilde{\beta}= {(\beta_1, \beta_2)}^{\top}\) in the role of \(\tilde{x}\). Let \(\tilde{a}(\tilde{\beta}) = {(\beta_1, \beta_1\beta_2)}^{\top}\) and \(\tilde{b}(\tilde{\beta}) = {(\beta_2, \beta_1)}^{\top}\). Then:
\[ \tilde{a} \cdot \tilde{b} = \beta_1 \cdot \beta_2 + \beta_1\beta_2 \cdot \beta_1 = \beta_1\beta_2 + \beta_1^2\beta_2 \]
By direct calculation:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}}(\tilde{a} \cdot \tilde{b})}_{2 \times 1} = \frac{\partial}{\partial \tilde{\beta}}(\beta_1\beta_2 + \beta_1^2\beta_2) = \begin{pmatrix} \beta_2 + 2\beta_1\beta_2 \\ \beta_1 + \beta_1^2 \end{pmatrix} \]
By the product rule (Theorem 3), using \(\underbrace{\frac{\partial}{\partial \tilde{\beta}}{\tilde{a}}^{\top}}_{2 \times 2} = \begin{pmatrix}1 & \beta_2 \\ 0 & \beta_1\end{pmatrix}\) and \(\underbrace{\frac{\partial}{\partial \tilde{\beta}}{\tilde{b}}^{\top}}_{2 \times 2} = \begin{pmatrix}0 & 1 \\ 1 & 0\end{pmatrix}\):
\[ \begin{aligned} \underbrace{\frac{\partial}{\partial \tilde{\beta}}(\tilde{a} \cdot \tilde{b})}_{2 \times 1} &= \underbrace{\begin{pmatrix}1 & \beta_2 \\ 0 & \beta_1\end{pmatrix}}_{2 \times 2} \underbrace{\begin{pmatrix}\beta_2 \\ \beta_1\end{pmatrix}}_{2 \times 1} + \underbrace{\begin{pmatrix}0 & 1 \\ 1 & 0\end{pmatrix}}_{2 \times 2} \underbrace{\begin{pmatrix}\beta_1 \\ \beta_1\beta_2\end{pmatrix}}_{2 \times 1} \\ &= \begin{pmatrix}\beta_2 + \beta_1\beta_2 \\ \beta_1^2\end{pmatrix} + \begin{pmatrix}\beta_1\beta_2 \\ \beta_1\end{pmatrix} \\ &= \begin{pmatrix}\beta_2 + 2\beta_1\beta_2 \\ \beta_1^2 + \beta_1\end{pmatrix} \end{aligned} \]
Both methods agree.
Theorem 4 (Derivative of a linear map) If \(\mathbf{A}\) is an \(m \times p\) matrix that is constant with respect to \(\tilde{\beta}\), then:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}} (\mathbf{A}\tilde{\beta})}_{p \times m} = \underbrace{{\mathbf{A}}^{\top}}_{p \times m} \]
Proof. For entry \((i,j)\), where row \(i\) indexes the denominator \(\tilde{\beta}\) (see Definition 3) and column \(j\) indexes the numerator \(\mathbf{A}\tilde{\beta}\):
\[ \begin{aligned} \left[\frac{\partial}{\partial \tilde{\beta}} (\mathbf{A}\tilde{\beta})\right]_{ij} &= \frac{\partial}{\partial \beta_i} (\mathbf{A}\tilde{\beta})_j \\ &= \frac{\partial}{\partial \beta_i} \sum_{k=1}^{p} a_{jk} \beta_k \\ &= a_{ji} \\ &= \left[{\mathbf{A}}^{\top}\right]_{ij} \end{aligned} \]
Example 6 (Derivative of a linear map) Let \(\mathbf{A} = \begin{pmatrix} 2 & 3 \end{pmatrix}\) (\(1 \times 2\)) and \(\tilde{\beta}= {(\beta_1, \beta_2)}^{\top}\). Then \(\mathbf{A}\tilde{\beta}= 2\beta_1 + 3\beta_2\), and by Theorem 4:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}}(\mathbf{A}\tilde{\beta})}_{2 \times 1} = \underbrace{{\mathbf{A}}^{\top}}_{2 \times 1} = \begin{pmatrix} 2 \\ 3 \end{pmatrix} \]
Theorem 5 (Vector-derivative of a matrix-vector product) If \(\mathbf{A}\) is an \(m \times q\) matrix that is constant with respect to \(\tilde{\beta}\), and \(\tilde{v} = \tilde{v}(\tilde{\beta})\) is a \(q \times 1\) vector that depends on the \(p \times 1\) vector \(\tilde{\beta}\), then:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}} (\mathbf{A}\tilde{v})}_{p \times m} = \underbrace{\mathopen{}\left(\frac{\partial}{\partial \tilde{\beta}} \tilde{v}\right)\mathclose{}}_{p \times q} \underbrace{{\mathbf{A}}^{\top}}_{q \times m} \]
Proof. For entry \((i,j)\), where row \(i\) indexes the denominator \(\tilde{\beta}\) and column \(j\) indexes the numerator \(\mathbf{A}\tilde{v}\) (see Definition 3):
\[ \begin{aligned} \left[\frac{\partial}{\partial \tilde{\beta}} (\mathbf{A}\tilde{v})\right]_{ij} &= \frac{\partial}{\partial \beta_i} (\mathbf{A}\tilde{v})_j \\ &= \frac{\partial}{\partial \beta_i} \sum_{k=1}^{q} a_{jk} v_k \\ &= \sum_{k=1}^{q} a_{jk} \frac{\partial}{\partial \beta_i} v_k \\ &= \sum_{k=1}^{q} \left[\frac{\partial}{\partial \tilde{\beta}} \tilde{v}\right]_{ik} \left[{\mathbf{A}}^{\top}\right]_{kj} \\ &= \left[\mathopen{}\left(\frac{\partial}{\partial \tilde{\beta}} \tilde{v}\right)\mathclose{} {\mathbf{A}}^{\top}\right]_{ij} \end{aligned} \]
Example 7 (Vector-derivative of a matrix-vector product) Let \(\mathbf{A} = \begin{pmatrix} 2 & 3 \end{pmatrix}\) (\(1 \times 2\), constant) and \(\tilde{v}(\tilde{\beta}) = {(\beta_1^2, \beta_2^2)}^{\top}\). Then \(\mathbf{A}\tilde{v} = 2\beta_1^2 + 3\beta_2^2\). By Theorem 5:
\[ \begin{aligned} \underbrace{\frac{\partial}{\partial \tilde{\beta}}(\mathbf{A}\tilde{v})}_{2 \times 1} &= \begin{pmatrix} 2\beta_1 & 0 \\ 0 & 2\beta_2 \end{pmatrix} \begin{pmatrix} 2 \\ 3 \end{pmatrix} \\ &= \begin{pmatrix} 4\beta_1 \\ 6\beta_2 \end{pmatrix} \end{aligned} \]
Theorem 6 (Vector-derivative of a product of matrices) If \(\mathbf{A}\) (\(\ell \times m\)) and \(\mathbf{B}\) (\(m \times q\)) are constant with respect to \(\tilde{\beta}\), and \(\tilde{v} = \tilde{v}(\tilde{\beta})\) is a \(q \times 1\) vector that depends on the \(p \times 1\) vector \(\tilde{\beta}\), then:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}} (\mathbf{A} \mathbf{B} \tilde{v})}_{p \times \ell} = \underbrace{\mathopen{}\left(\frac{\partial}{\partial \tilde{\beta}} \tilde{v}\right)\mathclose{}}_{p \times q} \underbrace{{\mathbf{B}}^{\top}}_{q \times m} \underbrace{{\mathbf{A}}^{\top}}_{m \times \ell} \]
Proof. Apply Theorem 5 with the constant \(\ell \times q\) matrix \(\mathbf{A}\mathbf{B}\), then use the transpose of a product:
\[ \begin{aligned} \frac{\partial}{\partial \tilde{\beta}} (\mathbf{A} \mathbf{B} \tilde{v}) &= \mathopen{}\left(\frac{\partial}{\partial \tilde{\beta}} \tilde{v}\right)\mathclose{} {(\mathbf{A}\mathbf{B})}^{\top} && \text{(derivative of a matrix-vector product)} \\ &= \mathopen{}\left(\frac{\partial}{\partial \tilde{\beta}} \tilde{v}\right)\mathclose{} {\mathbf{B}}^{\top}\,{\mathbf{A}}^{\top} && \text{(transpose of a product)} \end{aligned} \]
Example 8 Let \(\mathbf{A} = \begin{pmatrix}1 & 0\end{pmatrix}\) (\(1 \times 2\)), \(\mathbf{B} = \begin{pmatrix}2 & 0 \\ 0 & 3\end{pmatrix}\) (\(2 \times 2\)), and \(\tilde{v}(\tilde{\beta}) = \tilde{\beta}\) where \(\tilde{\beta}= {(\beta_1, \beta_2)}^{\top}\). Then \(\mathbf{A}\mathbf{B}\tilde{v} = 2\beta_1\), and:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}}(\mathbf{A}\mathbf{B}\tilde{v})}_{2 \times 1} = \underbrace{\mathopen{}\left(\frac{\partial}{\partial \tilde{\beta}}\tilde{\beta}\right)\mathclose{}}_{2 \times 2} \underbrace{{\mathbf{B}}^{\top}}_{2 \times 2} \underbrace{{\mathbf{A}}^{\top}}_{2 \times 1} = \mathbf{I}_2 \begin{pmatrix}2 & 0 \\ 0 & 3\end{pmatrix} \begin{pmatrix}1 \\ 0\end{pmatrix} = \begin{pmatrix}2 \\ 0\end{pmatrix} \]
Corollary 1 (Derivative of a dot product, transpose-product form) If \(\tilde{x}\) is constant with respect to \(\tilde{\beta}\), then:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}} (\underbrace{{\tilde{x}}^{\top}}_{1 \times p} \underbrace{\tilde{\beta}}_{p \times 1})}_{p \times 1} = \underbrace{\tilde{x}}_{p \times 1} \]
Proof. Using Theorem 2:
Since \({\tilde{x}}^{\top}\tilde{\beta}= \tilde{x}\cdot \tilde{\beta}\) (see dot product), and \(\tilde{x}\) is constant with respect to \(\tilde{\beta}\):
\[ \frac{\partial}{\partial \tilde{\beta}}({\tilde{x}}^{\top}\tilde{\beta}) = \frac{\partial}{\partial \tilde{\beta}}(\tilde{x}\cdot \tilde{\beta}) = \tilde{x} \]
by Theorem 2.
Proof. Using Theorem 5:
Since \(\tilde{x}\) is constant with respect to \(\tilde{\beta}\), \(\mathbf{A} = {\tilde{x}}^{\top}\) is a constant \(1 \times p\) matrix. Applying Theorem 5 with \(\tilde{v} = \tilde{\beta}\) (so \(\frac{\partial}{\partial \tilde{\beta}}\tilde{\beta}= \mathbf{I}\)):
\[ \begin{aligned} \frac{\partial}{\partial \tilde{\beta}}({\tilde{x}}^{\top}\tilde{\beta}) &= \mathopen{}\left(\frac{\partial}{\partial \tilde{\beta}}\tilde{\beta}\right)\mathclose{} {({\tilde{x}}^{\top})}^{\top} \\ &= \mathbf{I} \cdot \tilde{x}\\ &= \tilde{x} \end{aligned} \]
Example 9 (Derivative of a transpose product) Let \(\tilde{x}= {(3, 5)}^{\top}\) and \(\tilde{\beta}= {(\beta_1, \beta_2)}^{\top}\). Then \({\tilde{x}}^{\top}\tilde{\beta}= 3\beta_1 + 5\beta_2\), and by Corollary 1:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}}\left(\underbrace{{\tilde{x}}^{\top}}_{1 \times 2}\underbrace{\tilde{\beta}}_{2 \times 1}\right)}_{2 \times 1} = \underbrace{\tilde{x}}_{2 \times 1} = \begin{pmatrix} 3 \\ 5 \end{pmatrix} \]
Theorem 7 (Derivative of a quadratic form) For a quadratic form (see quadratic form), if \(\mathbf{S}\) is a symmetric \(p \times p\) matrix that is constant with respect to \(\tilde{\beta}\), then:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}} ({\tilde{\beta}}^{\top} \mathbf{S} \tilde{\beta})}_{p \times 1} = \underbrace{2 \mathbf{S} \tilde{\beta}}_{p \times 1} \]
Proof. Expanding entry-wise, \({\tilde{\beta}}^{\top} \mathbf{S} \tilde{\beta}= \sum_{j=1}^p \sum_{k=1}^p s_{jk} \beta_j \beta_k\). Differentiating component-wise with respect to \(\beta_i\) for \(i = 1, \ldots, p\):
\[ \begin{aligned} \left[\frac{\partial}{\partial \tilde{\beta}}({\tilde{\beta}}^{\top}\mathbf{S}\tilde{\beta})\right]_i &= \frac{\partial}{\partial \beta_i} \sum_{j=1}^p \sum_{k=1}^p s_{jk} \beta_j \beta_k && \text{(expand quadratic form)} \\ &= \sum_{k=1}^p s_{ik} \beta_k + \sum_{j=1}^p s_{ji} \beta_j && \text{(product rule for } \beta_i \beta_k \text{)} \\ &= [\mathbf{S}\tilde{\beta}]_i + [{\mathbf{S}}^{\top}\tilde{\beta}]_i && \text{(matrix-vector multiplication definition)} \\ &= [(\mathbf{S} + {\mathbf{S}}^{\top})\tilde{\beta}]_i && \text{(linearity of matrix multiplication)} \end{aligned} \]
When \(\mathbf{S}\) is symmetric (\(\mathbf{S} = {\mathbf{S}}^{\top}\)), \(\mathbf{S} + {\mathbf{S}}^{\top} = 2\mathbf{S}\), so \(\frac{\partial}{\partial \tilde{\beta}}({\tilde{\beta}}^{\top}\mathbf{S}\tilde{\beta}) = 2\mathbf{S}\tilde{\beta}\).
Example 10 (Derivative of a quadratic form) Let \(\mathbf{S} = \begin{pmatrix} 3 & 1 \\ 1 & 2 \end{pmatrix}\) (\(2 \times 2\), symmetric and constant) and \(\tilde{\beta}= {(\beta_1, \beta_2)}^{\top}\). Then \({\tilde{\beta}}^{\top}\mathbf{S}\tilde{\beta}= 3\beta_1^2 + 2\beta_1\beta_2 + 2\beta_2^2\). By Theorem 7:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}}({\tilde{\beta}}^{\top}\mathbf{S}\tilde{\beta})}_{2 \times 1} = 2 \mathbf{S} \tilde{\beta} = 2 \begin{pmatrix} 3 & 1 \\ 1 & 2 \end{pmatrix} \begin{pmatrix} \beta_1 \\ \beta_2 \end{pmatrix} = \begin{pmatrix} 6\beta_1 + 2\beta_2 \\ 2\beta_1 + 4\beta_2 \end{pmatrix} \]
Differentiating component-wise directly:
\[ \begin{pmatrix} \frac{\partial}{\partial \beta_1}(3\beta_1^2 + 2\beta_1\beta_2 + 2\beta_2^2) \\ \frac{\partial}{\partial \beta_2}(3\beta_1^2 + 2\beta_1\beta_2 + 2\beta_2^2) \end{pmatrix} = \begin{pmatrix} 6\beta_1 + 2\beta_2 \\ 2\beta_1 + 4\beta_2 \end{pmatrix} \]
Both methods agree.
Corollary 2 (Derivative of a simple quadratic form) \[ \underbrace{\frac{\partial}{\partial \tilde{\beta}} ({\tilde{\beta}}^{\top}\tilde{\beta})}_{p \times 1} = \underbrace{2\tilde{\beta}}_{p \times 1} \]
Proof. Applying Theorem 7 with \(\mathbf{S} = \mathbf{I}_{p \times p}\) (which is symmetric and constant with respect to \(\tilde{\beta}\)):
\[ \begin{aligned} \frac{\partial}{\partial \tilde{\beta}}({\tilde{\beta}}^{\top}\tilde{\beta}) &= \frac{\partial}{\partial \tilde{\beta}}({\tilde{\beta}}^{\top}\mathbf{I}_{p \times p}\tilde{\beta}) && \text{(rewrite with identity matrix)} \\ &= 2\mathbf{I}_{p \times p}\tilde{\beta} && \text{(derivative of a quadratic form, with } \mathbf{S} = \mathbf{I}_{p \times p} \text{)} \\ &= 2\tilde{\beta} && \text{(identity matrix property)} \end{aligned} \]
Example 11 (Derivative of a sum of squares) Let \(\tilde{\beta}= {(\beta_1, \beta_2)}^{\top}\), so \({\tilde{\beta}}^{\top}\tilde{\beta}= \beta_1^2 + \beta_2^2\). By Corollary 2:
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}}({\tilde{\beta}}^{\top}\tilde{\beta})}_{2 \times 1} = 2\tilde{\beta} = \begin{pmatrix} 2\beta_1 \\ 2\beta_2 \end{pmatrix} \]
Direct partial differentiation yields the same column vector.
Theorem 8 (Vector chain rule) Let \(\tilde{x}\) be a \(p \times 1\) vector, let \(\tilde{y}= \tilde{g}(\tilde{x})\) be a \(q \times 1\) vector-valued function of \(\tilde{x}\), and let \(z = f(\tilde{y})\) be a scalar-valued function of \(\tilde{y}\), where \(\tilde{g}\) and \(f\) have continuous partial derivatives. Then \(z = f(\tilde{g}(\tilde{x}))\), as a function of \(\tilde{x}\), satisfies
\[ \underbrace{\frac{\partial z}{\partial \tilde{x}}}_{p \times 1} = \underbrace{\frac{\partial \tilde{y}}{\partial \tilde{x}}}_{p \times q} \underbrace{\frac{\partial z}{\partial \tilde{y}}}_{q \times 1} \]
where \(\frac{\partial \tilde{y}}{\partial \tilde{x}}\) is the derivative of Definition 3 and \(\frac{\partial z}{\partial \tilde{x}}\) and \(\frac{\partial z}{\partial \tilde{y}}\) are vector derivatives (Definition 1).
Corollary 3 (Vector chain rule for quadratic forms) If \(\tilde{\varepsilon}= \tilde{\varepsilon}(\tilde{\beta})\) is an \(n \times 1\) vector-valued function of the \(p \times 1\) vector \(\tilde{\beta}\), with continuous partial derivatives, then
\[ \underbrace{\frac{\partial}{\partial \tilde{\beta}}{\mathopen{}\left(\tilde{\varepsilon}(\tilde{\beta})\cdot \tilde{\varepsilon}(\tilde{\beta})\right)\mathclose{}}}_{p \times 1} = \underbrace{\mathopen{}\left(\frac{\partial}{\partial \tilde{\beta}}\tilde{\varepsilon}(\tilde{\beta})\right)\mathclose{}}_{p \times n} \underbrace{\mathopen{}\left(2 \tilde{\varepsilon}(\tilde{\beta})\right)\mathclose{}}_{n \times 1} \]
Proof. Apply Theorem 8 with \(\tilde{x}= \tilde{\beta}\), \(\tilde{y}= \tilde{\varepsilon}\), \(q = n\), and \(z = f(\tilde{\varepsilon}) = \tilde{\varepsilon}\cdot \tilde{\varepsilon}= {\tilde{\varepsilon}}^{\top}\tilde{\varepsilon}\):
\[ \begin{aligned} \frac{\partial}{\partial \tilde{\beta}}\mathopen{}\left(\tilde{\varepsilon}\cdot \tilde{\varepsilon}\right)\mathclose{} &= \mathopen{}\left(\frac{\partial}{\partial \tilde{\beta}}\tilde{\varepsilon}\right)\mathclose{} \frac{\partial}{\partial \tilde{\varepsilon}}\mathopen{}\left({\tilde{\varepsilon}}^{\top}\tilde{\varepsilon}\right)\mathclose{} && \text{(vector chain rule)} \\ &= \mathopen{}\left(\frac{\partial}{\partial \tilde{\beta}}\tilde{\varepsilon}\right)\mathclose{} \mathopen{}\left(2\tilde{\varepsilon}\right)\mathclose{} && \text{(derivative of a simple quadratic form, in } \tilde{\varepsilon}\text{)} \end{aligned} \]
The second step is Corollary 2, applied to the \(n \times 1\) vector \(\tilde{\varepsilon}\) in place of \(\tilde{\beta}\).
Example 12 (Derivative of the residual sum of squares) Let \(\tilde{y}\) (\(n \times 1\)) and \(\mathbf{X}\) (\(n \times p\)) be constant with respect to \(\tilde{\beta}\), and let \(\tilde{\varepsilon}(\tilde{\beta}) = \tilde{y}- \mathbf{X}\tilde{\beta}\) be the vector of residuals. By Theorem 4, \(\frac{\partial}{\partial \tilde{\beta}}(\mathbf{X}\tilde{\beta}) = {\mathbf{X}}^{\top}\), and \(\frac{\partial}{\partial \tilde{\beta}}\tilde{y}= \mathbf{0}_{p \times n}\) because \(\tilde{y}\) is constant, so \(\frac{\partial}{\partial \tilde{\beta}}\tilde{\varepsilon}= -{\mathbf{X}}^{\top}\) (\(p \times n\)). By Corollary 3:
\[ \begin{aligned} \frac{\partial}{\partial \tilde{\beta}}\mathopen{}\left(\tilde{\varepsilon}\cdot \tilde{\varepsilon}\right)\mathclose{} &= \mathopen{}\left(-{\mathbf{X}}^{\top}\right)\mathclose{} \mathopen{}\left(2\tilde{\varepsilon}\right)\mathclose{} && \text{(vector chain rule for quadratic forms)} \\ &= -2\,{\mathbf{X}}^{\top}\mathopen{}\left(\tilde{y}- \mathbf{X}\tilde{\beta}\right)\mathclose{} && \text{(substitute } \tilde{\varepsilon}= \tilde{y}- \mathbf{X}\tilde{\beta}\text{)} \end{aligned} \]
Setting this \(p \times 1\) vector to \(\tilde{0}\) gives the normal equations \({\mathbf{X}}^{\top}\mathbf{X}\tilde{\beta}= {\mathbf{X}}^{\top}\tilde{y}\) of least squares.
Definition 5 (Matrix derivative) For a scalar-valued function \(f(\mathbf{X})\) of an \(m \times n\) matrix \(\mathbf{X}\), the matrix derivative is the \(m \times n\) matrix whose \((i,j)\) entry is the partial derivative of \(f\) with respect to the \((i,j)\) entry of \(\mathbf{X}\):
\[ \left[\frac{\partial}{\partial \mathbf{X}} f\right]_{ij} = \frac{\partial}{\partial X_{ij}} f \]
Example 13 (The matrix derivative of a trace) Let \(\mathbf{X}\) be a \(2 \times 2\) matrix and \(f(\mathbf{X}) = \operatorname{tr}(\mathbf{X}) = X_{11} + X_{22}\) (see trace). Then \(\frac{\partial}{\partial X_{ij}} f = 1\) if \(i = j\) and \(0\) otherwise, so:
\[ \frac{\partial}{\partial \mathbf{X}} f = \mathbf{I}_2 \]
Theorem 9 (Matrix derivative of the trace of a matrix product) If \(\mathbf{A}\) (\(r \times m\)) and \(\mathbf{B}\) (\(n \times r\)) are constant with respect to the \(m \times n\) matrix \(\mathbf{X}\), then:
\[ \underbrace{\frac{\partial}{\partial \mathbf{X}} \operatorname{tr}(\mathbf{A} \mathbf{X} \mathbf{B})}_{m \times n} = \underbrace{{\mathbf{A}}^{\top}}_{m \times r} \underbrace{{\mathbf{B}}^{\top}}_{r \times n} \]
Proof. Write \(A_{kl}\), \(X_{kl}\), and \(B_{kl}\) for the entries of \(\mathbf{A}\), \(\mathbf{X}\), and \(\mathbf{B}\). For entry \((i,j)\):
\[ \begin{aligned} \left[\frac{\partial}{\partial \mathbf{X}} \operatorname{tr}(\mathbf{A} \mathbf{X} \mathbf{B})\right]_{ij} &= \frac{\partial}{\partial X_{ij}} \sum_{a=1}^{r} \sum_{b=1}^{m} \sum_{c=1}^{n} A_{ab} X_{bc} B_{ca} && \text{(trace of the } r \times r \text{ product } \mathbf{A}\mathbf{X}\mathbf{B} \text{)} \\ &= \sum_{a=1}^{r} A_{ai} B_{ja} && \text{(only the terms with } b = i,\ c = j \text{ depend on } X_{ij} \text{)} \\ &= \sum_{a=1}^{r} \left[{\mathbf{A}}^{\top}\right]_{ia} \left[{\mathbf{B}}^{\top}\right]_{aj} && \text{(definition of the transpose)} \\ &= \left[{\mathbf{A}}^{\top}\,{\mathbf{B}}^{\top}\right]_{ij} && \text{(definition of matrix multiplication)} \end{aligned} \]
Example 14 (Differentiating a weighted trace) Let \(\mathbf{A} = \mathbf{I}_2\) (\(2 \times 2\)) and \(\mathbf{B} = \begin{pmatrix}2 & 0 \\ 0 & 3\end{pmatrix}\) (\(2 \times 2\)). Then \(\operatorname{tr}(\mathbf{A} \mathbf{X} \mathbf{B}) = 2X_{11} + 3X_{22}\), and:
\[ \underbrace{\frac{\partial}{\partial \mathbf{X}} \operatorname{tr}(\mathbf{A} \mathbf{X} \mathbf{B})}_{2 \times 2} = \underbrace{{\mathbf{A}}^{\top}}_{2 \times 2} \underbrace{{\mathbf{B}}^{\top}}_{2 \times 2} = \mathbf{I}_2 \begin{pmatrix}2 & 0 \\ 0 & 3\end{pmatrix} = \begin{pmatrix}2 & 0 \\ 0 & 3\end{pmatrix} \]
See also the Linear Algebra and Vector Calculus references.