Expectation

Last modified: 2026-11-07 04:59:05 (PST)

Definition 1 (Expectation, expected value, population mean) The expectation, expected value, or population mean of a discrete random variable \(X\), denoted \({\mathbb{E}\mathopen{}\left[X\right]\mathclose{}}\), \(\mu(X)\), or \(\mu_X\), is the mean of \(X\)’s possible values [1], weighted by the probability mass function of those values:

\[{\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} \stackrel{\text{def}}{=}\sum_{x \in \mathcal{R}(X)} x \cdot \mathbb{P}(X=x)\]

The expectation of a continuous random variable \(X\) with probability density function \(\mathbb{p}(X=x)\) is the mean of \(X\)’s possible values, weighted by that density:

\[{\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} \stackrel{\text{def}}{=}\int_{x\in \mathcal{R}(X)} x \cdot \mathbb{p}(X=x)\,dx\]

In both cases, the expectation is defined only when the sum or integral converges absolutely, that is, when it is finite with \(\mathopen{}\left|x\right|\mathclose{}\) in place of \(x\).

Remark 1 (Expectation is a probability-weighted average). The expectation of a random variable \(X\) is a weighted average of the values of \(X\), and the probabilities are the weights.

• A discrete random variable has a sum over its possible values. Each value \(x\) is multiplied by its weight \(\mathbb{P}(X=x)\), and the products are added. The weights are non-negative and add to \(1\).
• A continuous random variable has an integral over its possible values. Each value \(x\) is multiplied by its weight, the density \(\mathbb{p}(X=x)\), and the products are accumulated. The density is non-negative and integrates to \(1\).

The mean of \(n\) numbers \(y_1, \ldots, y_n\) is the special case with every weight equal to \(1/n\):

\[\frac{1}{n}\sum_{i=1}^ny_i = \sum_{i=1}^ny_i \cdot\frac{1}{n}\]

A value with a larger probability counts for more in the expectation. The expectation is the balance point of the possible values: the point where a ruler would balance if each value \(x\) carried a weight proportional to \(\mathbb{P}(X=x)\).

Example 1 (Expectation of a fair die roll) Let \(Y\) be the result of a fair die roll, so the range is \(\mathcal{R}(Y) ={\mathopen{}\left\{1, 2, 3, 4, 5, 6\right\}\mathclose{}}\) and \(\mathbb{P}(Y=y) = 1/6\) for each \(y\) in the range. Every value has the same weight, so the expectation is the ordinary mean of \(1, \ldots, 6\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} &= \sum_{y \in {\mathopen{}\left\{1, 2, 3, 4, 5, 6\right\}\mathclose{}}} y \cdot\mathbb{P}(Y=y) && \text{(definition of expectation for discrete r.v.)}\\ &= \mathopen{}\left(1 + 2 + 3 + 4 + 5 + 6\right)\mathclose{} \cdot\frac{1}{6} && \text{(every weight is } 1/6 \text{)}\\ &= \frac{21}{6} && \text{(add)}\\ &= 3.5 && \text{(divide)} \end{aligned} \]

Now load the die so that the result \(6\) has probability \(1/2\), and each of \(1, \ldots, 5\) has probability \(1/10\). The weights still add to \(1\), and the expectation moves toward \(6\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} &= \mathopen{}\left(1 + 2 + 3 + 4 + 5\right)\mathclose{} \cdot\frac{1}{10} + 6 \cdot\frac{1}{2} && \text{(weights } 1/10 \text{ and } 1/2 \text{)}\\ &= 1.5 + 3 && \text{(multiply)}\\ &= 4.5 && \text{(add)} \end{aligned} \]

Example 2 (The standard Cauchy distribution has no expectation) A random variable whose expectation is not defined is not exotic. Let \(X\) have the standard Cauchy distribution, with density \(\mathbb{p}(X=x) = \frac{1}{\pi(1 + x^2)}\) for every real \(x\). For \(x \ge 0\), \(\frac{1}{2\pi}\log(1 + x^2)\) is an antiderivative of \(\mathopen{}\left|x\right|\mathclose{} \cdot\frac{1}{\pi(1 + x^2)} = \frac{x}{\pi(1 + x^2)}\), and for \(x \le 0\), \(-\frac{1}{2\pi}\log(1 + x^2)\) is an antiderivative of \(\mathopen{}\left|x\right|\mathclose{} \cdot\frac{1}{\pi(1 + x^2)} = \frac{-x}{\pi(1 + x^2)}\). So, integrating over each half-line:

\[ \begin{aligned} \int_{0}^{\infty} \mathopen{}\left|x\right|\mathclose{} \cdot\frac{1}{\pi(1 + x^2)}\,dx &= \lim_{b \to \infty} \mathopen{}\left[\frac{1}{2\pi}\log(1 + x^2)\right]\mathclose{}_{0}^{b} && \text{(antiderivative on } [0, \infty) \text{)} \\ &= \lim_{b \to \infty} \frac{1}{2\pi}\log(1 + b^2) && \text{(evaluate at the bounds; } \log 1 = 0 \text{)} \\ &= \infty && \text{(} \log(1 + b^2) \to \infty \text{ as } b \to \infty \text{)} \end{aligned} \]

\[ \begin{aligned} \int_{-\infty}^{0} \mathopen{}\left|x\right|\mathclose{} \cdot\frac{1}{\pi(1 + x^2)}\,dx &= \lim_{a \to -\infty} \mathopen{}\left[-\frac{1}{2\pi}\log(1 + x^2)\right]\mathclose{}_{a}^{0} && \text{(antiderivative on } (-\infty, 0] \text{)} \\ &= \lim_{a \to -\infty} \frac{1}{2\pi}\log(1 + a^2) && \text{(evaluate at the bounds; } \log 1 = 0 \text{)} \\ &= \infty && \text{(} \log(1 + a^2) \to \infty \text{ as } a \to -\infty \text{)} \end{aligned} \]

Both halves diverge, so \(\int_{-\infty}^{\infty} \mathopen{}\left|x\right|\mathclose{} \cdot\frac{1}{\pi(1 + x^2)}\,dx\) diverges, and by Definition 1, \(X\) has no expectation.

Theorem 1 (Expectation of the Bernoulli distribution) If \(X \sim \operatorname{Ber}(\pi)\) (Bernoulli distribution), that is, \(\mathbb{P}(X = 1) = \pi\) and \(\mathbb{P}(X = 0) = 1 - \pi\), then:

\[{\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} = \pi\]

Proof. \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} &= \sum_{x\in \mathcal{R}(X)} x \cdot\mathbb{P}(X=x) && \text{(definition of expectation for discrete r.v.)} \\&= \sum_{x\in {\mathopen{}\left\{0,1\right\}\mathclose{}}} x \cdot\mathbb{P}(X=x) && \text{(range of Bernoulli r.v. is } \{0, 1\} \text{)} \\&= \mathopen{}\left(0 \cdot\mathbb{P}(X=0)\right)\mathclose{} + \mathopen{}\left(1 \cdot\mathbb{P}(X=1)\right)\mathclose{} && \text{(expand sum over } x = 0 \text{ and } x = 1 \text{)} \\&= \mathopen{}\left(0 \cdot(1-\pi)\right)\mathclose{} + \mathopen{}\left(1 \cdot\pi\right)\mathclose{} && \text{(substitute Bernoulli PMF values)} \\&= 0 + \pi&& \text{(multiplication by 0 and 1)} \\&= \pi&& \text{(addition of 0)} \end{aligned} \]

Theorem 2 (Expectation of time-to-event variables) If \(T\) is a non-negative random variable with survival function \(\mathbb{S}(t)\) and a defined expectation, then:

\[{\mathbb{E}\mathopen{}\left[T\right]\mathclose{}} = \int_{t=0}^{\infty} \mathbb{S}(t)\,dt\]

Proof. We prove the continuous case, in which \(T\) has a density \(\mathbb{f}\). The integrand \(\mathbb{f}(u) \cdot{\mathbb{1}\mathopen{}\left(0 \le t \le u\right)\mathclose{}}\) is non-negative on \([0, \infty) \times [0, \infty)\), so Tonelli’s theorem (the non-negative case of the Fubini–Tonelli theorem; Billingsley [2], Theorem 18.3) lets us exchange the order of integration:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[T\right]\mathclose{}} &= \int_{u=0}^{\infty} u\,\mathbb{f}(u)\,du && \text{(definition of expectation; } T \ge 0 \text{)}\\ &= \int_{u=0}^{\infty}\mathopen{}\left(\int_{t=0}^{u} 1\,dt\right)\mathclose{}\mathbb{f}(u)\,du && \text{(} u = \textstyle\int_0^u 1\,dt \text{)}\\ &= \int_{u=0}^{\infty}\int_{t=0}^{u} \mathbb{f}(u)\,dt\,du && \text{(move } \mathbb{f}(u) \text{ inside the inner integral)}\\ &= \int_{t=0}^{\infty}\int_{u=t}^{\infty} \mathbb{f}(u)\,du\,dt && \text{(Tonelli: exchange the order over } 0 \le t \le u \text{)}\\ &= \int_{t=0}^{\infty}\Pr(T>t)\,dt && \text{(integrate the density over } (t, \infty) \text{)}\\ &= \int_{t=0}^{\infty}\mathbb{S}(t)\,dt && \text{(definition of the survival function)} \end{aligned} \]

Every step also holds when \(\int_{u=0}^{\infty} u\,\mathbb{f}(u)\,du\) is infinite, because Tonelli’s theorem needs only a non-negative integrand; so \(\int_{t=0}^{\infty} \mathbb{S}(t)\,dt\) is finite exactly when \({\mathbb{E}\mathopen{}\left[T\right]\mathclose{}}\) is defined.

See [3] for an alternative presentation of this proof.

Example 3 (Mean of an exponential random variable via survival function) Let \(T\) be exponential with rate \({\lambda}> 0\), so \(\mathbb{S}(t) = \text{e}^{-{\lambda}t}\) for \(t \ge 0\), as computed on the random variables page. By Theorem 2:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[T\right]\mathclose{}} &= \int_0^\infty \mathbb{S}(t)\,dt && \text{(expectation via the survival function)}\\ &= \int_0^\infty \text{e}^{-{\lambda}t}\,dt && \text{(substitute the survival function)}\\ &= \mathopen{}\left[-\frac{1}{{\lambda}}\text{e}^{-{\lambda}t}\right]\mathclose{}_0^\infty && \text{(antiderivative)}\\ &= 0 - \mathopen{}\left(-\frac{1}{{\lambda}}\right)\mathclose{} && \text{(evaluate at the bounds)}\\ &= \frac{1}{{\lambda}} && \text{(simplify)} \end{aligned} \]

Definition 2 (Expectation of a random matrix) For a random matrix \(\mathbf{A}\) of size \(m \times n\) with \((i,j)\)-th element \(A_{ij}\), the expectation \(\mathbb{E}\mathbf{A}\) is the \(m \times n\) matrix whose \((i,j)\)-th element is \({\mathbb{E}\mathopen{}\left[A_{ij}\right]\mathclose{}}\):

\[ \mathbb{E}\mathbf{A} \stackrel{\text{def}}{=}\begin{pmatrix} {\mathbb{E}\mathopen{}\left[A_{11}\right]\mathclose{}} & {\mathbb{E}\mathopen{}\left[A_{12}\right]\mathclose{}} & \cdots & {\mathbb{E}\mathopen{}\left[A_{1n}\right]\mathclose{}} \\ {\mathbb{E}\mathopen{}\left[A_{21}\right]\mathclose{}} & {\mathbb{E}\mathopen{}\left[A_{22}\right]\mathclose{}} & \cdots & {\mathbb{E}\mathopen{}\left[A_{2n}\right]\mathclose{}} \\ \vdots & \vdots & \ddots & \vdots \\ {\mathbb{E}\mathopen{}\left[A_{m1}\right]\mathclose{}} & {\mathbb{E}\mathopen{}\left[A_{m2}\right]\mathclose{}} & \cdots & {\mathbb{E}\mathopen{}\left[A_{mn}\right]\mathclose{}} \end{pmatrix} \]

Remark. In other words, expectation is applied element-wise to a random matrix.

Example 4 (Expectation of a random vector of coin flips) For the two fair coin flips in the independence page’s example, the random vector \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= {(X_1, X_2)}^{\top}\) is a \(2 \times 1\) random matrix, and:

\[\mathbb{E}\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \begin{pmatrix}{\mathbb{E}\mathopen{}\left[X_1\right]\mathclose{}} \\ {\mathbb{E}\mathopen{}\left[X_2\right]\mathclose{}}\end{pmatrix} = \begin{pmatrix}1/2 \\ 1/2\end{pmatrix}\]

since each \(X_i\) is Bernoulli with \(\pi= 1/2\) (Theorem 1).

Theorem 3 (Law of the Unconscious Statistician (LOTUS)) Discrete case. For any function \(g\) of a discrete random variable \(X\) [4]:

\[{\mathbb{E}\mathopen{}\left[g(X)\right]\mathclose{}} = \sum_{x \in \mathcal{R}(X)} g(x) \cdot\mathbb{P}(X=x)\]

Continuous case. For any function \(g\) of a continuous random variable \(X\) with density \(\mathbb{p}(X=x)\):

\[{\mathbb{E}\mathopen{}\left[g(X)\right]\mathclose{}} = \int_{x \in \mathcal{R}(X)} g(x) \cdot\mathbb{p}(X=x)\, dx\]

Proof. We prove the discrete case. Let \(Y = g(X)\). For each \(y \in \mathcal{R}(Y)\), the event \(\{Y = y\}\) is the disjoint union of the events \(\{X = x\}\) over the values \(x\) with \(g(x) = y\), and each \(x \in \mathcal{R}(X)\) belongs to exactly one such group.

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[g(X)\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} && \text{(substitution } Y = g(X) \text{)} \\&= \sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{P}(Y=y) && \text{(definition of expectation for discrete r.v.)} \\&= \sum_{y \in \mathcal{R}(Y)} y \cdot\sum_{\substack{x \in \mathcal{R}(X) \\ g(x) = y}} \mathbb{P}(X=x) && \text{(countable additivity over the disjoint events } \{X = x\} \text{)} \\&= \sum_{y \in \mathcal{R}(Y)} \sum_{\substack{x \in \mathcal{R}(X) \\ g(x) = y}} y \cdot\mathbb{P}(X=x) && \text{(distribute } y \text{ into the inner sum)} \\&= \sum_{y \in \mathcal{R}(Y)} \sum_{\substack{x \in \mathcal{R}(X) \\ g(x) = y}} g(x) \cdot\mathbb{P}(X=x) && \text{(} y = g(x) \text{ for every term of the inner sum)} \\&= \sum_{x \in \mathcal{R}(X)} g(x) \cdot\mathbb{P}(X=x) && \text{(the groups partition } \mathcal{R}(X) \text{)} \end{aligned} \]

The last step regroups a possibly infinite sum, which is valid because \({\mathbb{E}\mathopen{}\left[g(X)\right]\mathclose{}}\) is defined only when the sum converges absolutely.

Remark. LOTUS says that to compute \({\mathbb{E}\mathopen{}\left[g(X)\right]\mathclose{}}\), we do not need to first find the distribution of \(g(X)\); we can compute the expectation directly using the distribution of \(X\). The continuous case is the density-weighted analogue of this argument, but a fully rigorous proof needs the general change-of-variables theorem for integrals against a pushforward measure — approximating \(g\) by simple functions and passing to the limit — which is beyond this page’s scope [2], [5], [6], [7].

Example 5 (Expected value of \(X^2\) for a Bernoulli variable) Let \(X \sim \operatorname{Ber}(\pi)\). By LOTUS (Theorem 3, discrete case):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X^2\right]\mathclose{}} &= \sum_{x \in {\mathopen{}\left\{0,1\right\}\mathclose{}}} x^2 \cdot\mathbb{P}(X=x) && \text{(LOTUS, discrete case)} \\ &= 0^2 \cdot\mathbb{P}(X=0) + 1^2 \cdot\mathbb{P}(X=1) && \text{(expand sum over } x = 0 \text{ and } x = 1 \text{)} \\ &= 0^2 \cdot(1-\pi) + 1^2 \cdot\pi && \text{(substitute Bernoulli PMF values)} \\ &= 0 + \pi && \text{(simplify each term)} \\ &= \pi && \text{(add)} \end{aligned} \]

Example 6 (Expected value of \(X^2\) for a Uniform(0,1) variable) Let \(X \sim \text{Uniform}(0,1)\) (uniform distribution), so \(\mathbb{p}(X=x) = 1\) for \(x \in [0,1]\). By LOTUS (Theorem 3, continuous case):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X^2\right]\mathclose{}} &= \int_0^1 x^2 \cdot\mathbb{p}(X=x)\,dx && \text{(LOTUS, continuous case)} \\&= \int_0^1 x^2 \cdot 1\,dx && \text{(}\mathbb{p}(X=x) = 1\text{ on } [0,1]\text{)} \\&= \mathopen{}\left[\frac{x^3}{3}\right]\mathclose{}_0^1 && \text{(antiderivative of } x^2\text{)} \\&= \frac{1}{3}. && \text{(evaluate at the bounds)} \end{aligned} \]

Theorem 4 (Linearity of expectation) For random variables \(X\) and \(Y\) with defined expectations, and constants \(a\), \(b\), and \(c\):

\[{\mathbb{E}\mathopen{}\left[aX + bY + c\right]\mathclose{}} = a{\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} + b{\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} + c\]

Proof. We prove the discrete case, applying Theorem 3 to the pair \((X, Y)\), which is a single discrete random object with values \((x, y)\), and \(g(x, y) = ax + by + c\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[aX + bY + c\right]\mathclose{}} &= \sum_{x \in \mathcal{R}(X)}\sum_{y \in \mathcal{R}(Y)} (ax + by + c) \cdot\mathbb{P}(X=x, Y=y) && \text{(LOTUS for the pair } (X, Y) \text{)}\\ &= a\sum_{x \in \mathcal{R}(X)}\sum_{y \in \mathcal{R}(Y)} x \cdot\mathbb{P}(X=x, Y=y) + b\sum_{x \in \mathcal{R}(X)}\sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{P}(X=x, Y=y) + c\sum_{x \in \mathcal{R}(X)}\sum_{y \in \mathcal{R}(Y)} \mathbb{P}(X=x, Y=y) && \text{(distribute the sum over the terms; constants factor out)}\\ &= a\sum_{x \in \mathcal{R}(X)}\sum_{y \in \mathcal{R}(Y)} x \cdot\mathbb{P}(X=x, Y=y) + b\sum_{y \in \mathcal{R}(Y)}\sum_{x \in \mathcal{R}(X)} y \cdot\mathbb{P}(X=x, Y=y) + c\sum_{x \in \mathcal{R}(X)}\sum_{y \in \mathcal{R}(Y)} \mathbb{P}(X=x, Y=y) && \text{(exchange the order of summation in the second term)}\\ &= a\sum_{x \in \mathcal{R}(X)} x \sum_{y \in \mathcal{R}(Y)} \mathbb{P}(X=x, Y=y) + b\sum_{y \in \mathcal{R}(Y)} y \sum_{x \in \mathcal{R}(X)} \mathbb{P}(X=x, Y=y) + c \cdot 1 && \text{(factor } x \text{ and } y \text{ out of the inner sums; total probability is 1)}\\ &= a\sum_{x \in \mathcal{R}(X)} x \cdot\mathbb{P}(X=x) + b\sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{P}(Y=y) + c && \text{(marginalize: countable additivity)}\\ &= a{\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} + b{\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} + c && \text{(definition of expectation)} \end{aligned} \]

Splitting and exchanging the sums is valid because the sums converge absolutely, since \(\mathopen{}\left|ax + by + c\right|\mathclose{} \le \mathopen{}\left|a\right|\mathclose{}\mathopen{}\left|x\right|\mathclose{} + \mathopen{}\left|b\right|\mathclose{}\mathopen{}\left|y\right|\mathclose{} + \mathopen{}\left|c\right|\mathclose{}\) and \({\mathbb{E}\mathopen{}\left[X\right]\mathclose{}}\) and \({\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}}\) are defined.

Beyond the discrete case, this proof does not carry over by replacing sums with integrals. That replacement would need a joint density for \((X, Y)\), and some pairs have none: if \(X\) is continuous, the pair \((X, X)\) has no joint density. In general, expectation is an integral against the probability measure, and the theorem is the linearity of that (Lebesgue) integral [2], which is beyond these notes’ scope.

Example 7 (Expected total of two dice) Let \(X\) and \(Y\) be the results of two six-sided die rolls. Each has expectation

\[ \begin{aligned} \sum_{x=1}^{6} x \cdot\tfrac{1}{6} &= \tfrac{21}{6} \\ &= 3.5, \end{aligned} \]

so:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X + Y\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} + {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} && \text{(linearity of expectation with } a = 1, b = 1, c = 0 \text{)}\\ &= 3.5 + 3.5 && \text{(substitute)}\\ &= 7 && \text{(add)} \end{aligned} \]

This calculation does not need the rolls to be independent.

Corollary 1 (Expectation of a linear function of one random variable) For a random variable \(X\) with a defined expectation, and constants \(a\) and \(c\):

\[{\mathbb{E}\mathopen{}\left[aX + c\right]\mathclose{}} = a{\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} + c\]

Proof. For discrete \(X\), this identity is Theorem 4 with \(b = 0\). For continuous \(X\) with density \(\mathbb{p}(X=x)\), apply Theorem 3 with \(g(x) = ax + c\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[aX + c\right]\mathclose{}} &= \int_{x \in \mathcal{R}(X)} (ax + c) \cdot\mathbb{p}(X=x)\,dx && \text{(LOTUS, continuous case)} \\ &= a\int_{x \in \mathcal{R}(X)} x \cdot\mathbb{p}(X=x)\,dx + c\int_{x \in \mathcal{R}(X)} \mathbb{p}(X=x)\,dx && \text{(linearity of the integral)} \\ &= a{\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} + c\int_{x \in \mathcal{R}(X)} \mathbb{p}(X=x)\,dx && \text{(definition of expectation)} \\ &= a{\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} + c \cdot 1 && \text{(a density integrates to 1)} \\ &= a{\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} + c && \text{(simplify)} \end{aligned} \]

Splitting the integral is valid because both pieces converge absolutely: the first because \({\mathbb{E}\mathopen{}\left[X\right]\mathclose{}}\) is defined, and the second because its integrand is a non-negative density.

Example 8 (Expectation of a rescaled uniform variable) Let \(X \sim \text{Uniform}(0,1)\), so

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} &= \int_0^1 x\,dx \\ &= \tfrac{1}{2}. \end{aligned} \]

By Corollary 1 with \(a = 2\) and \(c = 1\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[2X + 1\right]\mathclose{}} &= 2{\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} + 1 && \text{(expectation of a linear function)} \\ &= 2 \cdot\tfrac{1}{2} + 1 && \text{(substitute } {\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} = \tfrac{1}{2} \text{)} \\ &= 2 && \text{(evaluate)} \end{aligned} \]

Exercise 1 (Expected value of a density) Let \(X\) be a continuous random variable with density \(\mathbb{p}(X=x)\). The density evaluated at \(X\) is itself a random variable, \(\mathbb{p}(X=X)\).

Use Theorem 3 to write \({\mathbb{E}\mathopen{}\left[\mathbb{p}(X=X)\right]\mathclose{}}\) as an integral of the density.

Solution 1. Apply Theorem 3 to a continuous random variable, with \(g(x) = \mathbb{p}(X=x)\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[\mathbb{p}(X=X)\right]\mathclose{}} &= \int_{x \in \mathcal{R}(X)} \mathbb{p}(X=x) \cdot\mathbb{p}(X=x)\,dx && \text{(LOTUS, continuous case, with } g(x) = \mathbb{p}(X=x) \text{)} \\&= \int_{x \in \mathcal{R}(X)} \mathopen{}\left[\mathbb{p}(X=x)\right]\mathclose{}^2\,dx. && \text{(combine the two factors)} \end{aligned} \]

The expectation of the density is the integral of the squared density. It is large when the density is concentrated, because then \(X\) tends to land where the density is high.

Exercise 2 (Expected value of a uniform density) By Solution 1, the expectation of the density of a continuous random variable \(X\) is \(\int_{x \in \mathcal{R}(X)} \mathopen{}\left[\mathbb{p}(X=x)\right]\mathclose{}^2\,dx\).

Evaluate \({\mathbb{E}\mathopen{}\left[\mathbb{p}(X=X)\right]\mathclose{}}\) when \(X \sim \text{Uniform}(a, b)\) (uniform distribution).

Solution 2. The density is \(\mathbb{p}(X=x) = \tfrac{1}{b-a}\) on \([a, b]\) and \(0\) elsewhere:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[\mathbb{p}(X=X)\right]\mathclose{}} &= \int_a^b \mathopen{}\left(\frac{1}{b-a}\right)\mathclose{}^2\,dx && \text{(integral of the squared density; the density is } 0 \text{ outside } [a, b] \text{)} \\&= \frac{1}{(b-a)^2} \cdot(b-a) && \text{(the integrand does not depend on } x \text{, so the integral is the integrand times the interval length } b-a \text{)} \\&= \frac{1}{b-a}. && \text{(cancel one factor of } b-a \text{)} \end{aligned} \]

The density takes the single value \(\tfrac{1}{b-a}\) on \([a, b]\), so its expectation is that same value.

Exercise 3 (Expected value of a normal density) By Solution 1, the expectation of the density of a continuous random variable \(X\) is \(\int_{x \in \mathcal{R}(X)} \mathopen{}\left[\mathbb{p}(X=x)\right]\mathclose{}^2\,dx\).

Evaluate \({\mathbb{E}\mathopen{}\left[\mathbb{p}(X=X)\right]\mathclose{}}\) when \(X \sim {\operatorname{N}\mathopen{}\left(\mu, \sigma^2\right)\mathclose{}}\) (normal distribution), using the integral \(\int_{-\infty}^{\infty} {\operatorname{exp}\mathopen{}\left\{-u^2\right\}\mathclose{}}\,du = \sqrt{\pi}\).

Solution 3. The density is \(\mathbb{p}(X=x) = \frac{1}{\sqrt{2\pi\sigma^2}}{\operatorname{exp}\mathopen{}\left\{-\frac{(x-\mu)^2}{2\sigma^2}\right\}\mathclose{}}\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[\mathbb{p}(X=X)\right]\mathclose{}} &= \int_{-\infty}^{\infty} \frac{1}{2\pi\sigma^2}{\operatorname{exp}\mathopen{}\left\{-\frac{(x-\mu)^2}{\sigma^2}\right\}\mathclose{}}\,dx && \text{(integral of the squared density; square the density)} \\&= \frac{1}{2\pi\sigma^2} \int_{-\infty}^{\infty} {\operatorname{exp}\mathopen{}\left\{-u^2\right\}\mathclose{}}\,\sigma\,du && \text{(substitute } u = (x-\mu)/\sigma \text{, so } dx = \sigma\,du \text{)} \\&= \frac{\sigma\sqrt{\pi}}{2\pi\sigma^2} && \text{(the given integral)} \\&= \frac{1}{2\sigma\sqrt{\pi}}. && \text{(simplify)} \end{aligned} \]

A larger standard deviation \(\sigma\) spreads the density out and lowers its expected value.

Lemma 1 (Linearity of expectation for finite sums) For random variables \(X_1, \ldots, X_n\) with defined expectations, and constants \(a_1, \ldots, a_n\):

\[{\mathbb{E}\mathopen{}\left[\sum_{i=1}^na_i X_i\right]\mathclose{}} = \sum_{i=1}^na_i {\mathbb{E}\mathopen{}\left[X_i\right]\mathclose{}}\]

Proof. Prove this by induction on \(n\), applying linearity of expectation for two variables in each step.

For \(n = 1\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[a_1 X_1\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[a_1 X_1 + 0 \cdot X_1\right]\mathclose{}} && \text{(} 0 \cdot X_1 = 0 \text{)} \\ &= {\mathbb{E}\mathopen{}\left[a_1 X_1 + 0 \cdot X_1 + 0\right]\mathclose{}} && \text{(} v + 0 = v \text{)} \\ &= a_1 {\mathbb{E}\mathopen{}\left[X_1\right]\mathclose{}} + 0 \cdot{\mathbb{E}\mathopen{}\left[X_1\right]\mathclose{}} + 0 && \text{(linearity of expectation, } a = a_1, b = 0, c = 0 \text{)} \\ &= a_1 {\mathbb{E}\mathopen{}\left[X_1\right]\mathclose{}} + 0 && \text{(} 0 \cdot v = 0 \text{)} \\ &= a_1 {\mathbb{E}\mathopen{}\left[X_1\right]\mathclose{}} && \text{(} v + 0 = v \text{)} \end{aligned} \]

For the inductive step, assume the lemma holds for \(n - 1\) variables, and let \(W \stackrel{\text{def}}{=}\sum_{i=1}^{n-1} a_i X_i\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[\sum_{i=1}^na_i X_i\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[W + a_n X_n\right]\mathclose{}} && \text{(split off the last term)} \\ &= {\mathbb{E}\mathopen{}\left[1 \cdot W + a_n X_n\right]\mathclose{}} && \text{(} 1 \cdot W = W \text{)} \\ &= {\mathbb{E}\mathopen{}\left[1 \cdot W + a_n X_n + 0\right]\mathclose{}} && \text{(} v + 0 = v \text{)} \\ &= 1 \cdot{\mathbb{E}\mathopen{}\left[W\right]\mathclose{}} + a_n {\mathbb{E}\mathopen{}\left[X_n\right]\mathclose{}} + 0 && \text{(linearity of expectation, } a = 1, b = a_n, c = 0 \text{)} \\ &= {\mathbb{E}\mathopen{}\left[W\right]\mathclose{}} + a_n {\mathbb{E}\mathopen{}\left[X_n\right]\mathclose{}} + 0 && \text{(} 1 \cdot v = v \text{)} \\ &= {\mathbb{E}\mathopen{}\left[W\right]\mathclose{}} + a_n {\mathbb{E}\mathopen{}\left[X_n\right]\mathclose{}} && \text{(} v + 0 = v \text{)} \\ &= \sum_{i=1}^{n-1} a_i {\mathbb{E}\mathopen{}\left[X_i\right]\mathclose{}} + a_n {\mathbb{E}\mathopen{}\left[X_n\right]\mathclose{}} && \text{(inductive hypothesis)} \\ &= \sum_{i=1}^na_i {\mathbb{E}\mathopen{}\left[X_i\right]\mathclose{}} && \text{(include the last term in the sum)} \end{aligned} \]

Example 9 (Expected total of five dice) Let \(X_1, \ldots, X_5\) be the outcomes of five rolls of a fair die. Each roll has expectation \({\mathbb{E}\mathopen{}\left[X_i\right]\mathclose{}} = 3.5\). Applying Lemma 1 with every \(a_i = 1\) gives the expected total:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[\sum_{i=1}^5 X_i\right]\mathclose{}} &= \sum_{i=1}^5 {\mathbb{E}\mathopen{}\left[X_i\right]\mathclose{}} && \text{(linearity of expectation for finite sums)} \\ &= 5 \cdot 3.5 && \text{(each roll has expectation } 3.5 \text{)} \\ &= 17.5 && \text{(multiply)} \end{aligned} \]

Exercise 4 (Why the sample mean is a random variable) Let \(X_1, \ldots, X_n\) be independent draws from the same distribution, each with population mean \(\mu\) and variance \(\sigma^2\), and let \(\bar X = \frac{1}{n}\sum_{i=1}^nX_i\) be their sample mean.

Use Theorem 4 to show that \(\mathbb{E}[\bar X] = \mu\), and say in one sentence what that does not tell us.

Solution 4. Pull the constant \(1/n\) out and distribute the sum over the terms, both by Theorem 4:

\[ \begin{aligned} \mathbb{E}[\bar X] &= \mathbb{E}\mathopen{}\left[\frac{1}{n}\sum_{i=1}^nX_i\right]\mathclose{} \\ &= \frac{1}{n}\sum_{i=1}^n\mathbb{E}[X_i] \\ &= \frac{1}{n}\,(n\mu) \\ &= \mu \end{aligned} \]

So the sample mean is right on average. Independence was never used: Theorem 4 holds regardless, so this much would be true even for draws that influence one another.

What it does not tell us is how far any particular sample mean falls from \(\mu\) — that is a statement about variance, not expectation, and it is where independence does the work. The distinction matters because an estimator or fitted model is a function of one particular sample. Knowing that the procedure is right on average says nothing about the estimate actually in front of you, and separating those two statements is foundational to understanding bias and variance.

Theorem 5 (The expectation of a sample mean) Let \(X_1, \ldots, X_n\) be random variables, with \(n \ge 1\), that each have finite population mean \(\mu\), and let \(\bar X \stackrel{\text{def}}{=}\frac{1}{n}\sum_{i=1}^nX_i\). The \(X_i\) need not be independent. Then

\[\mathbb{E}[\bar X] = \mu\]

Proof. This is the solution to Exercise 4, worked in Solution 4. It follows from Theorem 4, which needs no independence and applies because each \(\mathbb{E}[X_i] = \mu\) is finite.

1 Conditional distributions and expectations

Definition 3 (Conditional probability mass function) Let \(X\) and \(Y\) be jointly distributed discrete random variables. The conditional probability mass function of \(Y\) given \(X = x\) (for values of \(x\) with \(\mathbb{P}(X = x) > 0\)) is:

\[\mathbb{P}(Y = y \mid X = x) \stackrel{\text{def}}{=}\frac{\mathbb{P}(X = x,\, Y = y)}{\mathbb{P}(X = x)}\]

Example 10 (Conditional PMF from a vaccine trial) The vaccine dataset in the dobson package (Dobson and Barnett [8], Table 9.6) records the responses in a flu vaccine trial: \(X\) is treatment group (placebo or vaccine) and \(Y\) is the level of response to treatment (small, moderate, or large).

response_levels <- c("small", "moderate", "large")
vaccine_tab <-
  dobson::vaccine |>
  dplyr::mutate(response = factor(response, levels = response_levels)) |>
  xtabs(frequency ~ treatment + response, data = _)
pander::pander(as.data.frame.matrix(vaccine_tab))

	small	moderate	large
placebo	25	8	5
vaccine	6	18	11

n_vaccine <- sum(vaccine_tab)
n_placebo <- vaccine_tab["placebo", ]

Choose one of the \(n = 73\) trial participants at random (a random choice), and let \((X, Y)\) be that participant’s treatment and response. The joint PMF of \((X, Y)\) is then the table of frequencies divided by the total \(n\) trial participants. By the marginal PMF theorem, the marginal probability \(\mathbb{P}(X = \text{placebo})\) is:

\[ \begin{aligned} \mathbb{P}(X = \text{placebo}) &= \mathbb{P}(X = \text{placebo},\, Y = \text{small}) + \mathbb{P}(X = \text{placebo},\, Y = \text{moderate}) + \mathbb{P}(X = \text{placebo},\, Y = \text{large}) && \text{(marginal PMF from the joint PMF)} \\&= \tfrac{25}{73} + \tfrac{8}{73} + \tfrac{5}{73} && \text{(read the table)} \\&= \tfrac{38}{73} && \text{(add)} \end{aligned} \]

By Definition 3, the conditional PMF of \(Y\) given \(X = \text{placebo}\) is:

\[ \begin{aligned} \mathbb{P}(Y = \text{small} \mid X = \text{placebo}) &= \frac{\mathbb{P}(X = \text{placebo},\, Y = \text{small})}{\mathbb{P}(X = \text{placebo})} && \text{(definition of conditional PMF)} \\&= \frac{25/73}{38/73} && \text{(substitute)} \\&= \frac{25}{38} && \text{(cancel } 73 \text{)} \end{aligned} \]

Definition 4 (Conditional probability density function) Let \(X\) and \(Y\) be jointly distributed continuous random variables with joint density \(\mathbb{p}(X = x,\, Y = y)\) and marginal density \(\mathbb{p}(X = x)\). The conditional probability density function of \(Y\) given \(X = x\) (for values of \(x\) with \(\mathbb{p}(X = x) > 0\)) is:

\[\mathbb{p}(Y = y \mid X = x) \stackrel{\text{def}}{=}\frac{\mathbb{p}(X = x,\, Y = y)}{\mathbb{p}(X = x)}\]

Example 11 (Conditional PDF from a bivariate normal model of birthweight data) The birthweight dataset in the dobson package (Dobson and Barnett [8], Table 2.3) records gestational age (weeks) and birthweight (grams) for 12 boys and 12 girls. Let \(X\) be gestational age and \(Y\) be birthweight, pooling both sexes into \(n = 24\) observations.

birthweight <- dobson::birthweight
ga <- c(
  birthweight[["boys gestational age"]],
  birthweight[["girls gestational age"]]
)
wt <- c(birthweight[["boys weight"]], birthweight[["girls weight"]])
mu_x <- mean(ga)
sigma_x <- sd(ga)
mu_y <- mean(wt)
sigma_y <- sd(wt)
rho <- cor(ga, wt)
beta <- cov(ga, wt) / var(ga)
intercept <- mu_y - beta * mu_x
x0 <- 40
marg_dens_x0 <- dnorm(x0, mean = mu_x, sd = sigma_x)
cond_mean <- intercept + beta * x0
cond_sd <- sigma_y * sqrt(1 - rho^2)
tibble::tibble(
  parameter = c("mu_x", "sigma_x", "mu_y", "sigma_y", "rho"),
  value = round(c(mu_x, sigma_x, mu_y, sigma_y, rho), 4)
) |>
  pander::pander()

parameter	value
mu_x	38.54
sigma_x	1.817
mu_y	2968
sigma_y	282.1
rho	0.7443

Modeling \((X, Y)\) as bivariate normal with parameters set equal to these values computed from the data (they stand in for the parameters here): \(\mu_X\) and \(\mu_Y\) are the means of \(X\) and \(Y\), \(\sigma^2_X\) and \(\sigma^2_Y\) their variances, and \(\rho\) their correlation (the next page defines the last two). The joint density is (a standard result; e.g. Casella and Berger [6]):

\[ \mathbb{p}(X=x,\,Y=y) = \frac{1}{2\pi\sigma_X\sigma_Y\sqrt{1-\rho^2}} \text{e}^{-\frac{1}{2(1-\rho^2)} \mathopen{}\left[\frac{(x-\mu_X)^2}{\sigma^2_X} - \frac{2\rho(x-\mu_X)(y-\mu_Y)}{\sigma_X\sigma_Y} + \frac{(y-\mu_Y)^2}{\sigma^2_Y}\right]\mathclose{}} \]

A further standard fact about the bivariate normal (Casella and Berger [6]) is that the marginal distribution of \(X\) is normal, \(X \sim {\operatorname{N}\mathopen{}\left(\mu_X, \sigma^2_X\right)\mathclose{}}\). At \(x = 40\) weeks (a full-term pregnancy), \(\mu_X = 38.5417\) and \(\sigma_X = 1.8173\), so:

\[ \begin{aligned} \mathbb{p}(X=40) &= \frac{1}{\sigma_X\sqrt{2\pi}} \text{e}^{-\frac{(40-\mu_X)^2}{2\sigma^2_X}} && \text{(normal density at } x = 40 \text{)} \\&= \frac{1}{1.8173\sqrt{2\pi}} \text{e}^{-\frac{(40-38.5417)^2}{2(1.8173)^2}} && \text{(substitute } \mu_X \text{ and } \sigma_X \text{)} \\&\approx 0.1591 && \text{(evaluate)} \end{aligned} \]

By Definition 4, dividing the joint density by this marginal density and simplifying the exponent (completing the square in \(y\); Casella and Berger [6]) gives the conditional PDF of \(Y\) given \(X = 40\), which is itself normal with mean shifted along the regression line and variance reduced by a factor of \(1-\rho^2\):

\[ \begin{aligned} \mathbb{p}(Y=y \mid X=40) &= \frac{\mathbb{p}(X=40,\,Y=y)}{\mathbb{p}(X=40)} && \text{(definition of the conditional PDF)} \\&= \frac{1}{\sigma_Y\sqrt{2\pi(1-\rho^2)}} \text{e}^{-\frac{1}{2(1-\rho^2)}\mathopen{}\left[\frac{(40-\mu_X)^2}{\sigma^2_X} - \frac{2\rho(40-\mu_X)(y-\mu_Y)}{\sigma_X\sigma_Y} + \frac{(y-\mu_Y)^2}{\sigma^2_Y}\right]\mathclose{} + \frac{(40-\mu_X)^2}{2\sigma^2_X}} && \text{(substitute both densities; combine prefactors and exponents)} \\&= \frac{1}{\sigma_Y\sqrt{2\pi(1-\rho^2)}} \text{e}^{-\frac{\rho^2(40-\mu_X)^2}{2\sigma^2_X(1-\rho^2)} + \frac{\rho(40-\mu_X)(y-\mu_Y)}{\sigma_X\sigma_Y(1-\rho^2)} - \frac{(y-\mu_Y)^2}{2\sigma^2_Y(1-\rho^2)}} && \text{(combine the } (40-\mu_X)^2 \text{ terms: } 1 - \tfrac{1}{1-\rho^2} = \tfrac{-\rho^2}{1-\rho^2} \text{)} \\&= \frac{1}{\sigma_Y\sqrt{2\pi(1-\rho^2)}} \text{e}^{-\frac{1}{2\sigma^2_Y(1-\rho^2)} \mathopen{}\left[\rho^2\frac{\sigma^2_Y}{\sigma^2_X}(40-\mu_X)^2 - 2\rho\frac{\sigma_Y}{\sigma_X}(40-\mu_X)(y-\mu_Y) + (y-\mu_Y)^2\right]\mathclose{}} && \text{(factor } -\tfrac{1}{2\sigma^2_Y(1-\rho^2)} \text{ out of the exponent)} \\&= \frac{1}{\sigma_Y\sqrt{2\pi(1-\rho^2)}} \text{e}^{-\frac{\mathopen{}\left(y - \mathopen{}\left[\mu_Y + \rho\frac{\sigma_Y}{\sigma_X}(40-\mu_X)\right]\mathclose{}\right)\mathclose{}^2}{2\sigma_Y^2(1-\rho^2)}} && \text{(the bracket is a perfect square in } y \text{)} \end{aligned} \]

Multiplying out each of the last two exponents reproduces the one before it.

So \(Y \mid X = 40 \sim {\operatorname{N}\mathopen{}\left(3136.15,\, 188.37^2\right)\mathclose{}}\): the conditional mean, 3136.15 g, is exactly the fitted regression line’s prediction at \(x=40\) (\(-1484.9846 + 115.5283 \times 40 = 3136.15\)), matching R’s lm(wt ~ ga) fit directly:

pander::pander(coef(lm(wt ~ ga)))

(Intercept)	ga
-1485	115.5

Definition 5 (Conditional expectation) Discrete case. Let \(X\) and \(Y\) be jointly distributed discrete random variables. The conditional expectation of \(Y\) given \(X = x\), using Definition 3, is [9]:

\[{\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}} \stackrel{\text{def}}{=}\sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{P}(Y = y \mid X = x)\]

Continuous case. Let \(X\) and \(Y\) be jointly distributed continuous random variables. The conditional expectation of \(Y\) given \(X = x\), using Definition 4, is:

\[{\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}} \stackrel{\text{def}}{=}\int_{y \in \mathcal{R}(Y)} y \cdot\mathbb{p}(Y = y \mid X = x)\, dy\]

Example 12 (Conditional expectation from real trial and birthweight data) Discrete case. Continuing Example 10, score the vaccine trial’s response levels \(\text{small} = 1\), \(\text{moderate} = 2\), \(\text{large} = 3\). The conditional PMF of \(Y\) given \(X = \text{placebo}\) (from Example 10) is:

\[ \begin{aligned} \mathbb{P}(Y = \text{small} \mid X = \text{placebo}) &= \tfrac{25}{38} \\ \mathbb{P}(Y = \text{moderate} \mid X = \text{placebo}) &= \tfrac{8}{38} \\ \mathbb{P}(Y = \text{large} \mid X = \text{placebo}) &= \tfrac{5}{38} \end{aligned} \]

so:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y \mid X = \text{placebo}\right]\mathclose{}} &= 1 \cdot\tfrac{25}{38} + 2 \cdot\tfrac{8}{38} + 3 \cdot\tfrac{5}{38} && \text{(definition of conditional expectation)} \\&= \frac{56}{38} && \text{(common denominator)} \\&\approx 1.47 && \text{(divide)} \end{aligned} \]

Continuous case. Continuing Example 11, \(Y \mid X = 40 \sim {\operatorname{N}\mathopen{}\left(3136.15,\, 188.37^2\right)\mathclose{}}\). The mean of a normal distribution is its location parameter (Casella and Berger [6]), so integrating \(y\) against this conditional density gives:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y \mid X = 40\right]\mathclose{}} &= \int_{-\infty}^{\infty} y \cdot\mathbb{p}(Y=y \mid X=40)\,dy && \text{(definition of conditional expectation)} \\&= 3136.15 \text{ g} && \text{(the mean of a normal distribution is its location parameter)} \end{aligned} \]

matching the fitted regression line’s prediction at \(x=40\) weeks exactly, as expected since the conditional mean of a bivariate normal is the linear regression of \(Y\) on \(X\).

Definition 6 (Conditional expectation: mixed case) Suppose exactly one of \(X, Y\) is discrete and the other is continuous, with joint density-mass function \(\mathbb{p}(X = x,\, Y = y)\).

\(X\) discrete, \(Y\) continuous. Here \(\mathbb{p}(X=x,\,Y=y)\) is, for each fixed \(x\), a probability density in \(y\), with \(\int_{y \in \mathcal{R}(Y)} \mathbb{p}(X=x,\,Y=y)\,dy = \mathbb{P}(X=x)\). The conditional PDF of \(Y\) given \(X = x\) (for values of \(x\) with \(\mathbb{P}(X = x) > 0\)) is:

\[\mathbb{p}(Y = y \mid X = x) \stackrel{\text{def}}{=}\frac{\mathbb{p}(X = x,\, Y = y)}{\mathbb{P}(X = x)}\]

and the conditional expectation of \(Y\) given \(X = x\) is:

\[{\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}} \stackrel{\text{def}}{=}\int_{y \in \mathcal{R}(Y)} y \cdot\mathbb{p}(Y = y \mid X = x)\, dy\]

\(X\) continuous, \(Y\) discrete. Here \(\mathbb{p}(X=x,\,Y=y)\) is, for each fixed \(y\), a probability density in \(x\), and \(\sum_{y \in \mathcal{R}(Y)} \mathbb{p}(X=x,\,Y=y) = \mathbb{p}(X=x)\) is a density of \(X\). The conditional PMF of \(Y\) given \(X = x\) (for values of \(x\) with \(\mathbb{p}(X = x) > 0\)) is:

\[\mathbb{P}(Y = y \mid X = x) \stackrel{\text{def}}{=}\frac{\mathbb{p}(X = x,\, Y = y)}{\mathbb{p}(X = x)}\]

and the conditional expectation of \(Y\) given \(X = x\) is:

\[{\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}} \stackrel{\text{def}}{=}\sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{P}(Y = y \mid X = x)\]

Example 13 (Conditional expectation, one discrete variable and one continuous variable) \(X\) discrete, \(Y\) continuous. The plasma dataset in the dobson package (Dobson and Barnett [8], Table 6.25) records plasma inorganic phosphate levels (mg/dL) one hour after a glucose tolerance test, for hyperinsulinemic obese (H-O) and control (C) participants. Let \(X\) be group and \(Y\) be phosphate level.

plasma_summary <-
  dobson::plasma |>
  dplyr::filter(Group %in% c("H-O", "C")) |>
  dplyr::summarise(
    .by = Group,
    n = dplyr::n(),
    mean = mean(phosphate),
    sd = sd(phosphate)
  )
group_stat <- function(group, stat) {
  plasma_summary[[stat]][plasma_summary[["Group"]] == group]
}
n_ho <- group_stat("H-O", "n")
n_c <- group_stat("C", "n")
mean_ho <- group_stat("H-O", "mean")
sd_ho <- group_stat("H-O", "sd")
mean_c <- group_stat("C", "mean")
sd_c <- group_stat("C", "sd")
pander::pander(plasma_summary)

Group	n	mean	sd
H-O	11	3.945	0.7776
C	12	2.783	0.4086

Modeling phosphate as approximately normal within each group, with parameters set equal to each group’s sample mean and SD, and \(\mathbb{P}(X = \text{H-O}) = 11/23\):

\[Y \mid X = \text{H-O} \sim {\operatorname{N}\mathopen{}\left(3.95,\, 0.78^2\right)\mathclose{}}\] \[Y \mid X = \text{C} \sim {\operatorname{N}\mathopen{}\left(2.78,\, 0.41^2\right)\mathclose{}}\]

By Definition 6, since the mean of a normal distribution is its location parameter:

\[{\mathbb{E}\mathopen{}\left[Y \mid X = \text{H-O}\right]\mathclose{}} = 3.95 \text{ mg/dL}, \qquad {\mathbb{E}\mathopen{}\left[Y \mid X = \text{C}\right]\mathclose{}} = 2.78 \text{ mg/dL}\]

\(X\) continuous, \(Y\) discrete. The senility dataset in the dobson package (Dobson and Barnett [8], Table 7.8) records, for 54 elderly people, a WAIS (Wechsler Adult Intelligence Scale) score and whether symptoms of senility were present. Let \(X\) be WAIS score and \(Y\) indicate senility symptoms.

senility_fit <- glm(s ~ x, data = dobson::senility, family = binomial)
b0 <- coef(senility_fit)[["(Intercept)"]]
b1 <- coef(senility_fit)[["x"]]
pander::pander(coef(senility_fit))

(Intercept)	x
2.404	-0.3235

Modeling \(\mathbb{P}(Y = 1 \mid X = x)\) with the fitted logistic regression:

\[\operatorname{logit}\mathbb{P}(Y = 1 \mid X = x) = 2.4040 - 0.3235\, x\]

By Definition 6, since \(Y \mid X=x\) is Bernoulli:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}} &= 0 \cdot\mathbb{P}(Y=0 \mid X=x) + 1 \cdot\mathbb{P}(Y=1 \mid X=x) && \text{(definition of conditional expectation, mixed case)} \\&= \mathbb{P}(Y=1 \mid X=x) && \text{(simplify)} \\&= \frac{1}{1 + \text{e}^{-(2.4040 - 0.3235\, x)}} && \text{(invert the logit)} \end{aligned} \]

senility_p10 <- predict(
  senility_fit,
  newdata = tibble::tibble(x = 10),
  type = "response"
)
senility_p15 <- predict(
  senility_fit,
  newdata = tibble::tibble(x = 15),
  type = "response"
)

At \(x = 10\): \({\mathbb{E}\mathopen{}\left[Y \mid X=10\right]\mathclose{}} = 0.3\). At \(x = 15\): \({\mathbb{E}\mathopen{}\left[Y \mid X=15\right]\mathclose{}} = 0.08\) — a lower WAIS score is associated with higher predicted probability of senility symptoms.

Definition 7 (Conditional expectation function) The conditional expectation function \({\mathbb{E}\mathopen{}\left[Y \mid X\right]\mathclose{}}\) is the function (and hence random variable) of \(X\) obtained by evaluating \({\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}}\) at \(X\); specifically, \({\mathbb{E}\mathopen{}\left[Y \mid X\right]\mathclose{}} = g(X)\) where \(g(x) \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}}\).

Example 14 (The conditional expectation function of the birthweight model) Continuing Example 11 and Example 12, \((X, Y)\) (gestational age, birthweight) is modeled as bivariate normal. The general form of the conditional mean derived in Example 11, evaluated at an arbitrary \(x\) instead of just \(x=40\), gives the conditional expectation function directly:

\[ \begin{aligned} g(x) &= \mu_Y + \rho\frac{\sigma_Y}{\sigma_X}(x-\mu_X) \\&= -1484.9846 + 115.5283\, x \end{aligned} \]

which is exactly the fitted regression line (the general algebraic identity \(\mu_Y + \rho\frac{\sigma_Y}{\sigma_X}(x-\mu_X) = \mathopen{}\left(\mu_Y - \rho\frac{\sigma_Y}{\sigma_X}\mu_X\right)\mathclose{} + \rho\frac{\sigma_Y}{\sigma_X}x\) is what makes the bivariate-normal conditional mean linear in \(x\)). As a check, evaluating at \(x = 40\) recovers Example 12’s result:

\[g(40) = -1484.9846 + 115.5283 \times 40 = 3136.15 \text{ g}\]

Definition 8 (Conditional expectation of a function of \(X\) and \(Y\)) Let \(X\) and \(Y\) be jointly distributed random variables, \(h\) a function of two arguments, and \(x\) a value at which the conditional PMF or PDF of \(Y\) given \(X = x\) is defined (Definition 3, Definition 4, or Definition 6). The conditional expectation of \(h(X, Y)\) given \(X = x\) is, for discrete \(Y\):

\[{\mathbb{E}\mathopen{}\left[h(X, Y) \mid X = x\right]\mathclose{}} \stackrel{\text{def}}{=}\sum_{y \in \mathcal{R}(Y)} h(x, y) \cdot\mathbb{P}(Y = y \mid X = x)\]

and for continuous \(Y\):

\[{\mathbb{E}\mathopen{}\left[h(X, Y) \mid X = x\right]\mathclose{}} \stackrel{\text{def}}{=}\int_{y \in \mathcal{R}(Y)} h(x, y) \cdot\mathbb{p}(Y = y \mid X = x)\,dy\]

when the sum or integral converges absolutely. Evaluating this function of \(x\) at \(X\) gives the random variable \({\mathbb{E}\mathopen{}\left[h(X, Y) \mid X\right]\mathclose{}}\).

Example 15 (The second flip, given the first) In the joint PMF of two coin flips, write \(X\) for the first flip and \(Y\) for the total number of heads. Given \(X = 1\), \(Y\) is \(1\) or \(2\) with conditional probability \(1/2\) each, and \(Y - X\) is the second flip. With \(h(x, y) = (y - x)^2\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[(Y - X)^2 \mid X = 1\right]\mathclose{}} &= (1 - 1)^2 \cdot\tfrac{1}{2} + (2 - 1)^2 \cdot\tfrac{1}{2} && \text{(definition, with } x = 1 \text{)} \\ &= 0 + \tfrac{1}{2} && \text{(evaluate each term)} \\ &= \tfrac{1}{2} && \text{(add)} \end{aligned} \]

Corollary 2 (With \(h(x, y) = y\), the general definition gives \({{\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}}}\)) For jointly distributed \(X\) and \(Y\) and \(h(x, y) = y\), Definition 8 gives the conditional expectation of \(Y\) given \(X = x\) from Definition 5 (when \(X\) and \(Y\) are both discrete or both continuous) or from Definition 6 (when one is discrete and the other continuous):

\[{\mathbb{E}\mathopen{}\left[h(X, Y) \mid X = x\right]\mathclose{}} = {\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}}\]

Proof. For discrete \(Y\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[h(X, Y) \mid X = x\right]\mathclose{}} &= \sum_{y \in \mathcal{R}(Y)} h(x, y) \cdot\mathbb{P}(Y = y \mid X = x) && \text{(conditional expectation of a function, discrete } Y \text{)} \\ &= \sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{P}(Y = y \mid X = x) && \text{(} h(x, y) = y \text{)} \\ &= {\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}} && \text{(conditional expectation, discrete } Y \text{)} \end{aligned} \]

The last step is Definition 5 when \(X\) is discrete and Definition 6 when \(X\) is continuous. For continuous \(Y\), the same steps hold with integrals over \(y\) in place of the sums and \(\mathbb{p}(Y = y \mid X = x)\) in place of \(\mathbb{P}(Y = y \mid X = x)\).

Theorem 6 (Conditional LOTUS, discrete case) Let \(X\) and \(Y\) be jointly distributed discrete random variables, \(h\) a function of two arguments, \(W \stackrel{\text{def}}{=}h(X, Y)\), and \(x\) a value with \(\mathbb{P}(X = x) > 0\). Then \(W\) is discrete, and \({\mathbb{E}\mathopen{}\left[h(X, Y) \mid X = x\right]\mathclose{}}\) from Definition 8 equals \({\mathbb{E}\mathopen{}\left[W \mid X = x\right]\mathclose{}}\) from Definition 5, with one sum converging absolutely exactly when the other does.

Proof. \(W\) is discrete because its range is contained in \({\mathopen{}\left\{h(u, y) : u \in \mathcal{R}(X),\, y \in \mathcal{R}(Y)\right\}\mathclose{}}\), which is countable. For each \(w \in \mathcal{R}(W)\), the event \({\mathopen{}\left\{X = x,\, W = w\right\}\mathclose{}}\) is the disjoint union of the events \({\mathopen{}\left\{X = x,\, Y = y\right\}\mathclose{}}\) over the values \(y \in \mathcal{R}(Y)\) with \(h(x, y) = w\), so:

\[ \begin{aligned} \mathbb{P}(W = w \mid X = x) &= \frac{\mathbb{P}(X = x,\, W = w)}{\mathbb{P}(X = x)} && \text{(conditional PMF of } W \text{)} \\ &= \frac{1}{\mathbb{P}(X = x)} \sum_{\substack{y \in \mathcal{R}(Y) \\ h(x, y) = w}} \mathbb{P}(X = x,\, Y = y) && \text{(countable additivity over the disjoint events } {\mathopen{}\left\{X = x,\, Y = y\right\}\mathclose{}} \text{)} \\ &= \sum_{\substack{y \in \mathcal{R}(Y) \\ h(x, y) = w}} \mathbb{P}(Y = y \mid X = x) && \text{(divide each term; conditional PMF of } Y \text{)} \end{aligned} \]

Every \(y \in \mathcal{R}(Y)\) with \(\mathbb{P}(Y = y \mid X = x) > 0\) has \(h(x, y) \in \mathcal{R}(W)\), because some outcome has \(X = x\) and \(Y = y\), and \(W = h(x, y)\) there; the other values of \(y\) contribute terms equal to \(0\). So the groups \({\mathopen{}\left\{y \in \mathcal{R}(Y) : h(x, y) = w\right\}\mathclose{}}\), over \(w \in \mathcal{R}(W)\), contain every \(y\) with a non-zero term, each exactly once:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[W \mid X = x\right]\mathclose{}} &= \sum_{w \in \mathcal{R}(W)} w \cdot\mathbb{P}(W = w \mid X = x) && \text{(conditional expectation of } W \text{)} \\ &= \sum_{w \in \mathcal{R}(W)} w \cdot\sum_{\substack{y \in \mathcal{R}(Y) \\ h(x, y) = w}} \mathbb{P}(Y = y \mid X = x) && \text{(the display above)} \\ &= \sum_{w \in \mathcal{R}(W)} \sum_{\substack{y \in \mathcal{R}(Y) \\ h(x, y) = w}} w \cdot\mathbb{P}(Y = y \mid X = x) && \text{(distribute } w \text{ into the inner sum)} \\ &= \sum_{w \in \mathcal{R}(W)} \sum_{\substack{y \in \mathcal{R}(Y) \\ h(x, y) = w}} h(x, y) \cdot\mathbb{P}(Y = y \mid X = x) && \text{(} w = h(x, y) \text{ for every term of the inner sum)} \\ &= \sum_{y \in \mathcal{R}(Y)} h(x, y) \cdot\mathbb{P}(Y = y \mid X = x) && \text{(the groups contain every non-zero term once)} \\ &= {\mathbb{E}\mathopen{}\left[h(X, Y) \mid X = x\right]\mathclose{}} && \text{(conditional expectation of a function of } X \text{ and } Y \text{)} \end{aligned} \]

The same steps with \(\mathopen{}\left|w\right|\mathclose{}\) and \(\mathopen{}\left|h(x, y)\right|\mathclose{}\) in place of \(w\) and \(h(x, y)\) regroup a sum of non-negative terms, which is valid whether or not it converges, so one sum converges absolutely exactly when the other does, and then the regrouping above is valid too.

Remark. Definition 8 is the conditional version of LOTUS: for discrete \(X\), it is LOTUS applied under the probability measure \(\Pr(\cdot \mid X = x)\), and by Theorem 6, results about \({\mathbb{E}\mathopen{}\left[W \mid X\right]\mathclose{}}\) apply to it. \(Y\) can also be a pair \((Y, Z)\), with the sum or integral taken over both.

Theorem 7 (Linearity of conditional expectation) For jointly distributed \(X\) and \(Y\), functions \(h_1\) and \(h_2\) whose conditional expectations given \(X = x\) are defined, and constants \(a\) and \(b\):

\[{\mathbb{E}\mathopen{}\left[a h_1(X, Y) + b h_2(X, Y) \mid X = x\right]\mathclose{}} = a{\mathbb{E}\mathopen{}\left[h_1(X, Y) \mid X = x\right]\mathclose{}} + b{\mathbb{E}\mathopen{}\left[h_2(X, Y) \mid X = x\right]\mathclose{}}\]

Evaluating both sides at \(X\) gives \({\mathbb{E}\mathopen{}\left[a h_1(X, Y) + b h_2(X, Y) \mid X\right]\mathclose{}} = a{\mathbb{E}\mathopen{}\left[h_1(X, Y) \mid X\right]\mathclose{}} + b{\mathbb{E}\mathopen{}\left[h_2(X, Y) \mid X\right]\mathclose{}}\).

Proof. For discrete \(Y\):

\[ \begin{aligned} &{\mathbb{E}\mathopen{}\left[a h_1(X, Y) + b h_2(X, Y) \mid X = x\right]\mathclose{}} \\&= \sum_{y \in \mathcal{R}(Y)} \mathopen{}\left(a h_1(x, y) + b h_2(x, y)\right)\mathclose{} \cdot\mathbb{P}(Y = y \mid X = x) && \text{(definition of conditional expectation)} \\ &= a \sum_{y \in \mathcal{R}(Y)} h_1(x, y) \cdot\mathbb{P}(Y = y \mid X = x) + b \sum_{y \in \mathcal{R}(Y)} h_2(x, y) \cdot\mathbb{P}(Y = y \mid X = x) && \text{(distribute the sum over the terms; factor out the constants)} \\ &= a{\mathbb{E}\mathopen{}\left[h_1(X, Y) \mid X = x\right]\mathclose{}} + b{\mathbb{E}\mathopen{}\left[h_2(X, Y) \mid X = x\right]\mathclose{}} && \text{(definition of conditional expectation)} \end{aligned} \]

Splitting the sum is valid because both sums converge absolutely. For continuous \(Y\), the same steps hold with integrals over \(y\) in place of the sums, by linearity of the integral: unlike Theorem 4, only one variable, \(y\), is integrated, against one conditional density.

Example 16 (Linearity, given the first flip) Continuing Example 15, given \(X = 1\), \(Y\) is \(1\) or \(2\) with probability \(1/2\) each, so \({\mathbb{E}\mathopen{}\left[Y \mid X = 1\right]\mathclose{}} = 3/2\) and

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y^2 \mid X = 1\right]\mathclose{}} &= (1 + 4)/2 \\ &= 5/2. \end{aligned} \]

By Theorem 7 with

\[ \begin{aligned} a &= b \\ &= 1: \end{aligned} \]

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y + Y^2 \mid X = 1\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[Y \mid X = 1\right]\mathclose{}} + {\mathbb{E}\mathopen{}\left[Y^2 \mid X = 1\right]\mathclose{}} && \text{(linearity of conditional expectation)} \\ &= \tfrac{3}{2} + \tfrac{5}{2} && \text{(substitute)} \\ &= 4 && \text{(add)} \end{aligned} \]

As a check, directly:

\[ \begin{aligned} (1 + 1) \cdot\tfrac{1}{2} + (2 + 4) \cdot\tfrac{1}{2} &= 1 + 3 \\ &= 4. \end{aligned} \]

Theorem 8 (A function of \(X\) factors out of a conditional expectation given \(X\)) For jointly distributed \(X\) and \(Y\), a function \(g\) of one argument, and a function \(h\) of two arguments whose conditional expectation given \(X = x\) is defined:

\[{\mathbb{E}\mathopen{}\left[g(X)\,h(X, Y) \mid X = x\right]\mathclose{}} = g(x) \cdot{\mathbb{E}\mathopen{}\left[h(X, Y) \mid X = x\right]\mathclose{}}\]

Evaluating both sides at \(X\) gives \({\mathbb{E}\mathopen{}\left[g(X)\,h(X, Y) \mid X\right]\mathclose{}} = g(X) \cdot{\mathbb{E}\mathopen{}\left[h(X, Y) \mid X\right]\mathclose{}}\). In particular, taking \(h = 1\) gives \({\mathbb{E}\mathopen{}\left[g(X) \mid X\right]\mathclose{}} = g(X)\).

Proof. For discrete \(Y\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[g(X)\,h(X, Y) \mid X = x\right]\mathclose{}} &= \sum_{y \in \mathcal{R}(Y)} g(x)\,h(x, y) \cdot\mathbb{P}(Y = y \mid X = x) && \text{(definition of conditional expectation)} \\ &= g(x) \sum_{y \in \mathcal{R}(Y)} h(x, y) \cdot\mathbb{P}(Y = y \mid X = x) && \text{(} g(x) \text{ does not depend on } y \text{)} \\ &= g(x) \cdot{\mathbb{E}\mathopen{}\left[h(X, Y) \mid X = x\right]\mathclose{}} && \text{(definition of conditional expectation)} \end{aligned} \]

For \(h = 1\), the remaining sum is \(\sum_{y \in \mathcal{R}(Y)} \mathbb{P}(Y = y \mid X = x) = 1\):

\[ \begin{aligned} \sum_{y \in \mathcal{R}(Y)} \mathbb{P}(Y = y \mid X = x) &= \sum_{y \in \mathcal{R}(Y)} \frac{\mathbb{P}(X = x,\, Y = y)}{\mathbb{P}(X = x)} && \text{(definition of the conditional PMF)} \\ &= \frac{\mathbb{P}(X = x)}{\mathbb{P}(X = x)} && \text{(marginal PMF from a joint PMF)} \\ &= 1 && \text{(divide)} \end{aligned} \]

For continuous \(Y\), the same steps hold with integrals over \(y\) in place of the sums; the conditional density integrates to 1 by the marginal density theorem (or, in the mixed case, by the joint density-mass function’s marginal identity).

Remark. This result is often summarized as “taking out what is known” (e.g., Ross [10], Section 5.6.4): given \(X = x\), any function of \(X\) is the known constant \(g(x)\).

Example 17 (Taking out the first flip) Continuing Example 16, with \(g(x) = x\) and \(h(x, y) = y\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[XY \mid X = 1\right]\mathclose{}} &= 1 \cdot{\mathbb{E}\mathopen{}\left[Y \mid X = 1\right]\mathclose{}} && \text{(a function of } X \text{ factors out)} \\ &= \tfrac{3}{2} && \text{(substitute } {\mathbb{E}\mathopen{}\left[Y \mid X = 1\right]\mathclose{}} = \tfrac{3}{2} \text{)} \end{aligned} \]

and

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[XY \mid X = 0\right]\mathclose{}} &= 0 \cdot{\mathbb{E}\mathopen{}\left[Y \mid X = 0\right]\mathclose{}} \\ &= 0. \end{aligned} \]

As a check, given \(X = 1\) the product \(XY\) equals \(Y\), which is \(1\) or \(2\) with probability \(1/2\) each.

Exercise 5 (Expectation of a sum, given a joint PMF) Let \((X, Y)\) be discrete with joint probability mass function:

	\(Y = 0\)	\(Y = 1\)
\(X = 0\)	\(0.2\)	\(0.3\)
\(X = 1\)	\(0.1\)	\(0.4\)

Compute \({\mathbb{E}\mathopen{}\left[X + Y\right]\mathclose{}}\).

Solution. Treating \((X, Y)\) as a single discrete random object taking one of the four values \((0,0)\), \((0,1)\), \((1,0)\), \((1,1)\), LOTUS (Theorem 3) applied to \(g(x, y) = x + y\) gives directly:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X + Y\right]\mathclose{}} &= \sum_{x \in \{0,1\}} \sum_{y \in \{0,1\}} (x + y)\,\mathbb{P}(X = x,\, Y = y) && \text{(LOTUS)} \\ &= (0{+}0)(0.2) + (0{+}1)(0.3) + (1{+}0)(0.1) + (1{+}1)(0.4) && \text{(expand the four terms)} \\ &= 0 + 0.3 + 0.1 + 0.8 && \text{(multiply)} \\ &= 1.2 && \text{(add)} \end{aligned} \]

As a check:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} &= 0(0.5) + 1(0.5) \\ &= 0.5 \end{aligned} \]

and

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} &= 0(0.3) + 1(0.7) \\ &= 0.7, \end{aligned} \]

so

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X + Y\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} + {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} \\ &= 1.2 \end{aligned} \]

by linearity of expectation.

x_labs <- c("X=0", "X=0", "X=1", "X=1")
y_labs <- c("Y=0", "Y=1", "Y=0", "Y=1")
probs <- c(0.2, 0.3, 0.1, 0.4)

plotly::plot_ly(
  x = ~y_labs, y = ~probs, color = ~x_labs,
  colors = c("steelblue", "tomato"),
  type = "bar"
) |>
  plotly::layout(
    barmode = "group",
    xaxis = list(title = "Value of Y"),
    yaxis = list(
      title = "Joint probability, P(X = x, Y = y)",
      range = c(0, 0.5)
    ),
    legend = list(title = list(text = "X value"))
  )

Figure 1: Joint probability mass function \(\mathbb{P}(X = x, Y = y)\). Marginal totals: \(\mathbb{P}(X = 0) = 0.5\), \(\mathbb{P}(X = 1) = 0.5\), \(\mathbb{P}(Y = 0) = 0.3\), \(\mathbb{P}(Y = 1) = 0.7\).

Remark. Note that \(X\) and \(Y\) are not independent here:

\[ \begin{aligned} \mathbb{P}(X = 0, Y = 0) &= 0.2 \\ &\neq 0.15 \\ &= \mathbb{P}(X = 0)\,\mathbb{P}(Y = 0). \end{aligned} \]

LOTUS applies regardless, since it requires only the actual joint mass function, not independence.

There are only four \((x,y)\) pairs here, so summing them in any order — row by row, column by column, or any other listing — gives the same total by ordinary commutativity and associativity of addition; no result about interchanging summation order is needed. Once the support is countably infinite, exchanging the order of summation needs a justification, which the Fubini–Tonelli theorem provides.

2 Fubini–Tonelli for expectations

Theorem 9 (Fubini–Tonelli theorem, measure-theoretic form) Let \(\mu_1\) and \(\mu_2\) be \(\sigma\)-finite measures on \(\sigma\)-algebras of subsets of sets \(S_1\) and \(S_2\), and let \(f\) be a measurable function on \(S_1 \times S_2\). If either

1. \(f \ge 0\) (Tonelli), or

2. \(\int_{S_1 \times S_2} \mathopen{}\left|f\right|\mathclose{}\,d(\mu_1 \otimes \mu_2) < \infty\) (Fubini),

then the integral of \(f\) against the product measure \(\mu_1 \otimes \mu_2\) equals both iterated integrals:

\[ \begin{aligned} \int_{S_1 \times S_2} f\,d(\mu_1 \otimes \mu_2) &= \int_{S_1}\mathopen{}\left(\int_{S_2} f(s_1, s_2)\,d\mu_2(s_2)\right)\mathclose{}\,d\mu_1(s_1) \\&= \int_{S_2}\mathopen{}\left(\int_{S_1} f(s_1, s_2)\,d\mu_1(s_1)\right)\mathclose{}\,d\mu_2(s_2). \end{aligned} \]

In case (b), each inner integral is finite except on a set of measure \(0\), which does not affect the outer integral.

Remark. For expectations, we use this measure-theoretic form of the Fubini–Tonelli theorem, which lets us exchange the order of integration (or summation) over a product of \(\sigma\)-finite measure spaces, provided the integrand is non-negative (Tonelli) or absolutely integrable (Fubini). Its proof is beyond these notes’ scope [2], Theorem 18.3. Lebesgue measure on the real line and counting measure on a countable set are both \(\sigma\)-finite, which gives the theorem a form stated in terms of a joint distribution.

Corollary 3 (Joint-distribution form (without independence; corollary of Fubini–Tonelli)) Let \((X, Y)\) be jointly distributed random variables whose joint distribution has a density \(f_{X,Y}\) with respect to a product of \(\sigma\)-finite reference measures \(\mu_X \otimes \mu_Y\) on \(\mathcal{R}(X) \times \mathcal{R}(Y)\), and let \(h : \mathcal{R}(X) \times \mathcal{R}(Y) \to \mathbb{R}\) be measurable. If either

1. \(h(X, Y) \ge 0\) almost surely, or

2. \({\mathbb{E}\mathopen{}\left[\mathopen{}\left|h(X, Y)\right|\mathclose{}\right]\mathclose{}} < \infty\),

then the expectation of \(h(X, Y)\) can be written as an iterated integral against \(f_{X,Y}\), with the order of integration exchangeable:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[h(X, Y)\right]\mathclose{}} &= \int_{\mathcal{R}(X)}\mathopen{}\left(\int_{\mathcal{R}(Y)} h(x, y)\,f_{X,Y}(x, y)\,d\mu_Y(y)\right)\mathclose{}\,d\mu_X(x) \\&= \int_{\mathcal{R}(Y)}\mathopen{}\left(\int_{\mathcal{R}(X)} h(x, y)\,f_{X,Y}(x, y)\,d\mu_X(x)\right)\mathclose{}\,d\mu_Y(y). \end{aligned} \]

The choice of reference measures covers three cases:

• Both continuous:

  \[ \begin{aligned} \mu_X &= \mu_Y \\ &= \text{Lebesgue measure}; \end{aligned} \]

  \(f_{X,Y}\) is the joint probability density function (PDF), and \(\int_{\mathcal{R}(X)} g(x)\,d\mu_X(x) = \int_{\mathcal{R}(X)} g(x)\,dx\).

• Both discrete:

  \[ \begin{aligned} \mu_X &= \mu_Y \\ &= \text{counting measure}; \end{aligned} \]

  \(f_{X,Y}(x,y) = \mathbb{P}(X = x,\, Y = y)\) is the joint probability mass function (PMF), and \(\int_{\mathcal{R}(X)} g(x)\,d\mu_X(x) = \sum_{x \in \mathcal{R}(X)} g(x)\).

• Mixed (one continuous, one discrete): one reference measure is Lebesgue and the other is counting; \(f_{X,Y}\) is the joint density-mass function, \(f_{X,Y}(x,y) = f_{X \mid Y}(x \mid y)\,\mathbb{P}(Y = y)\) (or \(\mathbb{P}(X = x \mid Y = y)\,f_Y(y)\) if \(X\) is discrete and \(Y\) continuous), and the iterated integrals combine an ordinary integral with a sum. The conditional densities/PMFs here are defined the same way as in Definition 6, just conditioning on \(Y\) instead of \(X\).

Proof. Apply Theorem 9 with \(\mu_1 = \mu_X\) and \(\mu_2 = \mu_Y\) to the integrand \(h(x,y)\,f_{X,Y}(x,y)\) on \(\mathcal{R}(X) \times \mathcal{R}(Y)\). Lebesgue measure and counting measure on a countable set are each \(\sigma\)-finite, so \(\mu_X \otimes \mu_Y\) is \(\sigma\)-finite in all three cases. The relevant condition is (a) when \(h \ge 0\) and (b) when \({\mathbb{E}\mathopen{}\left[\mathopen{}\left|h(X, Y)\right|\mathclose{}\right]\mathclose{}} < \infty\). Independence is not required. When \(X\) and \(Y\) are independent, \(f_{X,Y}(x,y) = f_X(x)\,f_Y(y)\) (or \(\mathbb{P}(X=x,Y=y) = \mathbb{P}(X=x)\,\mathbb{P}(Y=y)\) in the discrete case), and the iterated integrals factor into separate integrals over the marginals.

Example 18 (Expectation of a product of independent variables) Let \(X \sim \mathrm{Uniform}(0, 1)\) and \(Y \sim \mathrm{Uniform}(0, 2)\), independently distributed. Compute \({\mathbb{E}\mathopen{}\left[XY\right]\mathclose{}}\).

We apply Corollary 3 (both-continuous case) with \(h(x, y) = xy\). Since \(X\) and \(Y\) are independent with densities \(f_X(x) = 1\) on \([0,1]\) and \(f_Y(y) = \tfrac{1}{2}\) on \([0,2]\), the joint density factors as

\[ \begin{aligned} f_{X,Y}(x,y) &= f_X(x)\,f_Y(y) \\ &= \tfrac{1}{2}, \end{aligned} \]

and

\[ \begin{aligned} \mu_X &= \mu_Y \\ &= \text{Lebesgue measure}: \end{aligned} \]

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[XY\right]\mathclose{}} &= \int_0^1 \mathopen{}\left(\int_0^2 xy \cdot\tfrac{1}{2}\,dy\right)\mathclose{}\,dx && \text{(joint-distribution form of Fubini--Tonelli)} \\&= \int_0^1 x\mathopen{}\left(\frac{1}{2}\int_0^2 y\,dy\right)\mathclose{}\,dx && \text{(factor constants out of the inner integral)} \\&= \int_0^1 x \cdot\frac{1}{2} \cdot\mathopen{}\left[\frac{y^2}{2}\right]\mathclose{}_0^2\,dx && \text{(antiderivative of } y \text{)} \\&= \int_0^1 x \cdot\frac{1}{2} \cdot 2\,dx && \text{(evaluate at the bounds)} \\&= \int_0^1 x\,dx && \text{(simplify)} \\&= \frac{1}{2} && \text{(integrate)} \end{aligned} \]

As a check: \({\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} = \tfrac{1}{2}\), \({\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} = 1\), and \({\mathbb{E}\mathopen{}\left[X\right]\mathclose{}}{\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} = \tfrac{1}{2}\).

Example 19 (When independence fails: a counterexample) Correctly applying Corollary 3 requires the actual joint density \(f_{X,Y}\) — not the product of marginals \(f_X(x)\,f_Y(y)\), which is valid only when \(X\) and \(Y\) are independent. Using the wrong joint density gives the wrong answer.

Let \(X \sim \mathrm{Uniform}(0, 1)\) and set \(Y = X\) (so \(X\) and \(Y\) are perfectly correlated and not independent).

True expectation:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[XY\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[X \cdot X\right]\mathclose{}} && \text{(} Y = X \text{)} \\&= {\mathbb{E}\mathopen{}\left[X^2\right]\mathclose{}} && \text{(} X \cdot X = X^2 \text{)} \\&= \int_0^1 x^2\,dx && \text{(LOTUS, with the uniform density } 1 \text{ on } [0, 1] \text{)} \\&= \frac{1}{3} && \text{(integrate)} \end{aligned} \]

Erroneously applying the product-measure formula:

Note that Fubini–Tonelli’s own conditions still hold here (\(h(x,y) = xy\) is nonnegative and integrable), so the error is not a failure of Fubini–Tonelli itself. Rather, the error is using the wrong measure: the joint distribution of \((X, X)\) is concentrated on the diagonal \(\{(x, x) : x \in [0, 1]\} \subset [0, 1]^2\), which has Lebesgue measure zero in \(\mathbb{R}^2\). The joint distribution is therefore not absolutely continuous with respect to two-dimensional Lebesgue measure, so no joint density \(f_{X,Y}\) on \([0, 1]^2\) exists, which is the reference density Corollary 3 requires.

The following calculation is what someone would erroneously write if they assumed independence and used \(f_X(x)\,f_Y(y)\) as a “joint density” — a function that does not in fact correspond to the joint distribution of \((X, X)\). The marginals \(X \sim \mathrm{Uniform}(0,1)\) and \(Y \sim \mathrm{Uniform}(0,1)\) do have densities

\[ \begin{aligned} f_X &= f_Y \\ &= 1, \end{aligned} \]

but the product \(f_X(x)\,f_Y(y) = 1\) on \([0, 1]^2\) is the density of an independent pair, not of \((X, X)\):

\[ \begin{aligned} \int_0^1\!\int_0^1 xy \cdot f_X(x) \cdot f_Y(y)\,dy\,dx &= \int_0^1\!\int_0^1 xy\,dy\,dx && \text{(} f_X(x)\,f_Y(y) = 1 \text{ on } [0, 1]^2 \text{)} \\&= \int_0^1 x\mathopen{}\left(\int_0^1 y\,dy\right)\mathclose{}\,dx && \text{(factor } x \text{ out of the inner integral)} \\&= \int_0^1 x \cdot\frac{1}{2}\,dx && \text{(} \textstyle\int_0^1 y\,dy = \tfrac{1}{2} \text{)} \\&= \frac{1}{4} && \text{(} \textstyle\int_0^1 \tfrac{x}{2}\,dx = \tfrac{1}{4} \text{)} \end{aligned} \]

This calculation recovers \({\mathbb{E}\mathopen{}\left[XY\right]\mathclose{}}\) for independent uniforms (\(\tfrac{1}{4}\)), not \({\mathbb{E}\mathopen{}\left[XX\right]\mathclose{}}\) for the perfectly correlated pair (\(\tfrac{1}{3}\)). The lesson is that Corollary 3 requires the actual joint density \(f_{X,Y}\). For independent \((X, Y)\), this density factors as \(f_X(x)\,f_Y(y)\); for dependent \((X, Y)\), \(f_{X,Y}\) need not factor — and for \((X, X)\), no joint density on \(\mathbb{R}^2\) exists at all, so Corollary 3 simply does not apply.

set.seed(204)
n <- 400
x_dep <- runif(n)
y_dep <- x_dep
x_ind <- runif(n)
y_ind <- runif(n)

plotly::plot_ly() |>
  plotly::add_trace(
    type = "scatter", mode = "markers",
    x = x_ind, y = y_ind,
    name = "Assumed independent (X<sub>1</sub>, X<sub>2</sub>)",
    marker = list(size = 5, color = "#999999", opacity = 0.5)
  ) |>
  plotly::add_trace(
    type = "scatter", mode = "markers",
    x = x_dep, y = y_dep,
    name = "Actual (X, X) on diagonal",
    marker = list(size = 6, color = "#b40426")
  ) |>
  plotly::layout(
    xaxis = list(title = "Value of X", range = c(0, 1), scaleanchor = "y"),
    yaxis = list(title = "Value of Y", range = c(0, 1)),
    legend = list(orientation = "h", y = -0.2)
  )

Figure 2: Samples from the joint distribution of \((X, X)\) (red, on the diagonal) versus an independent pair \((X_1, X_2)\) with the same marginals (grey, scattered over \([0, 1]^2\)). The actual joint mass for \((X, X)\) is concentrated on a 1-dimensional diagonal — a set of Lebesgue measure zero in \(\mathbb{R}^2\) — so no joint density on \([0, 1]^2\) exists, and the “\(f_X(x)\,f_Y(y) = 1\)” calculation integrates against the wrong measure (the grey distribution).

Example 20 (Both-continuous case: joint PDF on a non-rectangular support) Let \((X, Y)\) have joint density \(f_{X,Y}(x, y) = 2\) for \(0 \le x \le y \le 1\) (and \(0\) otherwise). Compute \({\mathbb{E}\mathopen{}\left[X + Y\right]\mathclose{}}\).

By Corollary 3:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X + Y\right]\mathclose{}} &= \int_0^1\!\int_0^y (x + y) \cdot 2\,dx\,dy && \text{(joint-distribution form of Fubini--Tonelli, over the support } 0 \le x \le y \le 1 \text{)} \\&= 2\int_0^1 \mathopen{}\left[\frac{x^2}{2} + xy\right]\mathclose{}_{x=0}^{x=y}\,dy && \text{(antiderivative in } x \text{)} \\&= 2\int_0^1 \mathopen{}\left(\frac{y^2}{2} + y^2\right)\mathclose{}\,dy && \text{(evaluate at the bounds)} \\&= 2\int_0^1 \frac{3y^2}{2}\,dy && \text{(add)} \\&= 3\int_0^1 y^2\,dy && \text{(simplify the constant)} \\&= 3 \cdot\frac{1}{3} && \text{(integrate)} \\&= 1 && \text{(multiply)} \end{aligned} \]

n_grid <- 51
x_seq <- seq(0, 1, length.out = n_grid)
y_seq <- seq(0, 1, length.out = n_grid)

z_mat <- outer(x_seq, y_seq, function(x, y) {
  z <- rep(2, length(x))
  z[x > y] <- NA
  z
})

plotly::plot_ly(x = ~x_seq, y = ~y_seq, z = ~t(z_mat)) |>
  plotly::add_surface(showscale = FALSE) |>
  plotly::layout(scene = list(
    xaxis = list(title = "Value of X"),
    yaxis = list(title = "Value of Y"),
    zaxis = list(title = "Joint density, f(x, y)", range = c(0, 2.5)),
    camera = list(eye = list(x = 1.6, y = -1.6, z = 0.8))
  ))

Figure 3: Joint density \(f_{X,Y}(x, y) = 2\) on the triangular support \(\{(x, y) : 0 \le x \le y \le 1\}\), and zero elsewhere. The total “volume” under the density is \(2 \cdot \tfrac{1}{2} = 1\), as required.

Theorem 10 (Law of iterated expectations) For any two random variables \(X\) and \(Y\) with \({\mathbb{E}\mathopen{}\left[\mathopen{}\left|Y\right|\mathclose{}\right]\mathclose{}} < \infty\):

\[{\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} = {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y \mid X\right]\mathclose{}}\right]\mathclose{}}\]

Proof. Discrete case. When \(X\) and \(Y\) are discrete, applying Definition 1 to \({\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y \mid X\right]\mathclose{}}\right]\mathclose{}}\) and then the law of total probability applied to the countable partition \(\{X = x : x \in \mathcal{R}(X)\}\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y \mid X\right]\mathclose{}}\right]\mathclose{}} &= \sum_{x \in \mathcal{R}(X)} {\mathbb{E}\mathopen{}\left[Y \mid X=x\right]\mathclose{}} \cdot\mathbb{P}(X=x) && \text{(LOTUS for the function } x \mapsto {\mathbb{E}\mathopen{}\left[Y \mid X=x\right]\mathclose{}} \text{)} \\&= \sum_{x \in \mathcal{R}(X)} \mathopen{}\left(\sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{P}(Y=y \mid X=x)\right)\mathclose{} \cdot\mathbb{P}(X=x) && \text{(definition of conditional expectation } {\mathbb{E}\mathopen{}\left[Y \mid X=x\right]\mathclose{}} \text{)} \\&= \sum_{y \in \mathcal{R}(Y)} y \cdot\sum_{x \in \mathcal{R}(X)} \mathbb{P}(Y=y \mid X=x) \cdot\mathbb{P}(X=x) && \text{(exchange order of summation: Fubini, since } {\mathbb{E}\mathopen{}\left[\mathopen{}\left|Y\right|\mathclose{}\right]\mathclose{}} < \infty \text{)} \\&= \sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{P}(Y=y) && \text{(law of total probability over } X \text{)} \\&= {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} && \text{(definition of expectation of } Y \text{)} \end{aligned} \]

Continuous case. When \(X\) and \(Y\) are continuous:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y \mid X\right]\mathclose{}}\right]\mathclose{}} &= \int_{x \in \mathcal{R}(X)} {\mathbb{E}\mathopen{}\left[Y \mid X=x\right]\mathclose{}} \cdot\mathbb{p}(X=x)\, dx && \text{(LOTUS)} \\&= \int_{x \in \mathcal{R}(X)} \mathopen{}\left(\int_{y \in \mathcal{R}(Y)} y \cdot\mathbb{p}(Y=y \mid X=x)\, dy\right)\mathclose{} \cdot\mathbb{p}(X=x)\, dx && \text{(definition of conditional expectation)} \\&= \int_{x \in \mathcal{R}(X)} \int_{y \in \mathcal{R}(Y)} y \cdot\mathbb{p}(X=x, Y=y)\, dy\, dx && \text{(definition of conditional density: } \mathbb{p}(Y=y \mid X=x)\,\mathbb{p}(X=x) = \mathbb{p}(X=x, Y=y) \text{)} \\&= \int_{y \in \mathcal{R}(Y)} y \cdot\mathopen{}\left(\int_{x \in \mathcal{R}(X)} \mathbb{p}(X=x, Y=y)\, dx\right)\mathclose{}\, dy && \text{(Fubini's theorem, since } {\mathbb{E}\mathopen{}\left[\mathopen{}\left|Y\right|\mathclose{}\right]\mathclose{}} < \infty \text{)} \\&= \int_{y \in \mathcal{R}(Y)} y \cdot\mathbb{p}(Y=y)\, dy && \text{(marginal density from a joint density)} \\&= {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} && \text{(definition of expectation)} \end{aligned} \]

Mixed case, \(X\) discrete and \(Y\) continuous. With the joint density-mass function \(\mathbb{p}(X = x,\, Y = y)\), Corollary 3 applies with \(h(x, y) = y\), counting measure for \(X\), and Lebesgue measure for \(Y\); its condition (b) holds because \({\mathbb{E}\mathopen{}\left[\mathopen{}\left|Y\right|\mathclose{}\right]\mathclose{}} < \infty\). The sum runs over the \(x\) with \(\mathbb{P}(X = x) > 0\), where Definition 6 defines \({\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}}\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y \mid X\right]\mathclose{}}\right]\mathclose{}} &= \sum_{x \in \mathcal{R}(X)} {\mathbb{E}\mathopen{}\left[Y \mid X=x\right]\mathclose{}} \cdot\mathbb{P}(X=x) && \text{(LOTUS, discrete case)} \\&= \sum_{x \in \mathcal{R}(X)} \mathopen{}\left(\int_{\mathcal{R}(Y)} y \cdot\mathbb{p}(Y=y \mid X=x)\,dy\right)\mathclose{} \cdot\mathbb{P}(X=x) && \text{(definition of conditional expectation, mixed case)} \\&= \sum_{x \in \mathcal{R}(X)} \int_{\mathcal{R}(Y)} y \cdot\mathbb{p}(Y=y \mid X=x) \cdot\mathbb{P}(X=x)\,dy && \text{(move the constant } \mathbb{P}(X=x) \text{ inside the integral)} \\&= \sum_{x \in \mathcal{R}(X)} \int_{\mathcal{R}(Y)} y \cdot\mathbb{p}(X=x,\, Y=y)\,dy && \text{(definition of the conditional density, mixed case)} \\&= {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} && \text{(joint-distribution form of Fubini--Tonelli, with } h(x, y) = y \text{)} \end{aligned} \]

Mixed case, \(X\) continuous and \(Y\) discrete. The same steps apply with the integral and the sum swapped, over the \(x\) with \(\mathbb{p}(X = x) > 0\); at any other \(x\), \(\mathbb{p}(X = x,\, Y = y) = 0\) for every \(y\), because these non-negative terms sum to \(\mathbb{p}(X = x)\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y \mid X\right]\mathclose{}}\right]\mathclose{}} &= \int_{\mathcal{R}(X)} {\mathbb{E}\mathopen{}\left[Y \mid X=x\right]\mathclose{}} \cdot\mathbb{p}(X=x)\,dx && \text{(LOTUS, continuous case)} \\&= \int_{\mathcal{R}(X)} \mathopen{}\left(\sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{P}(Y=y \mid X=x)\right)\mathclose{} \cdot\mathbb{p}(X=x)\,dx && \text{(definition of conditional expectation, mixed case)} \\&= \int_{\mathcal{R}(X)} \sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{P}(Y=y \mid X=x) \cdot\mathbb{p}(X=x)\,dx && \text{(move the constant } \mathbb{p}(X=x) \text{ inside the sum)} \\&= \int_{\mathcal{R}(X)} \sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{p}(X=x,\, Y=y)\,dx && \text{(definition of the conditional PMF, mixed case)} \\&= {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} && \text{(joint-distribution form of Fubini--Tonelli, with } h(x, y) = y \text{)} \end{aligned} \]

Remark. Alternate names for this identity include: the tower rule, the tower property, the law of total expectation, and the smoothing theorem.

Example 21 (Fuel economy by number of cylinders) The Auto data record the number of engine cylinders and the fuel economy (miles per gallon) of 392 car models sold between 1970 and 1982. They come from the StatLib library at Carnegie Mellon University and ship with the book An Introduction to Statistical Learning, in the Python package ISLP (ISLP.load_data("Auto")) and the R package ISLR2 (ISLR2::Auto). This site has a copy at data/auto.csv, and both code tabs below read that file.

Choose one car at random, each with the same probability, and let \(X\) be its number of cylinders and \(Y\) its miles per gallon. Then \(\mathbb{P}(X = x)\) is the fraction of cars with \(x\) cylinders, and \({\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}}\) from Definition 5 is the average miles per gallon among those cars. The code below computes both for every \(x\), then checks Theorem 10 by averaging the conditional means with weights \(\mathbb{P}(X = x)\).

• Python
• R

import pandas as pd

auto = pd.read_csv("data/auto.csv")

by_cyl = auto.groupby("cylinders")["mpg"].agg(["count", "mean"])
by_cyl["P(X = x)"] = by_cyl["count"] / by_cyl["count"].sum()
by_cyl = by_cyl.rename(columns={"mean": "E(Y | X = x)"})
by_cyl = by_cyl[["count", "P(X = x)", "E(Y | X = x)"]]

lie_rhs = (by_cyl["P(X = x)"] * by_cyl["E(Y | X = x)"]).sum()
lie_lhs = auto["mpg"].mean()

by_cyl
#>            count  P(X = x)  E(Y | X = x)
#> cylinders                               
#> 3              4  0.010204     20.550000
#> 4            199  0.507653     29.283920
#> 5              3  0.007653     27.366667
#> 6             83  0.211735     19.973494
#> 8            103  0.262755     14.963107

auto <- read.csv("data/auto.csv")

counts <- table(auto$cylinders)
cond_mean <- tapply(auto$mpg, auto$cylinders, mean)
by_cyl <- data.frame(
  count = as.vector(counts),
  "P(X = x)" = as.vector(counts) / sum(counts),
  "E(Y | X = x)" = as.vector(cond_mean),
  row.names = names(counts),
  check.names = FALSE
)

lie_rhs <- sum(by_cyl[["P(X = x)"]] * by_cyl[["E(Y | X = x)"]])
lie_lhs <- mean(auto$mpg)

by_cyl

The conditional mean \({\mathbb{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}}\) is the function \(g(x)\) of Definition 7. Four-cylinder cars average 29.3 miles per gallon, and eight-cylinder cars average 15.0. Weighting the 5 conditional means by \(\mathbb{P}(X = x)\) gives 23.446, which equals the overall average miles per gallon, 23.446, as Theorem 10 states: \({\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y \mid X\right]\mathclose{}}\right]\mathclose{}} = {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}}\).

Theorem 11 (Conditional law of iterated expectations) For random variables \(X\), \(Y\), and \(Z\) with \({\mathbb{E}\mathopen{}\left[\mathopen{}\left|Y\right|\mathclose{}\right]\mathclose{}} < \infty\):

\[{\mathbb{E}\mathopen{}\left[Y \mid Z\right]\mathclose{}} = {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y \mid X,Z\right]\mathclose{}} \mid Z\right]\mathclose{}}\]

Proof. Choose a value \(z\) with positive probability (discrete case) or positive density (continuous case). Conditioning every probability on \(Z = z\) gives a probability measure, and the proof of Theorem 10 goes through under it, line by line.

Discrete case.

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y \mid X,Z\right]\mathclose{}} \mid Z=z\right]\mathclose{}} &= \sum_{x \in \mathcal{R}(X)} {\mathbb{E}\mathopen{}\left[Y \mid X=x,Z=z\right]\mathclose{}} \cdot\mathbb{P}(X=x \mid Z=z) && \text{(expectation given } Z=z \text{)} \\&= \sum_{x \in \mathcal{R}(X)} \mathopen{}\left(\sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{P}(Y=y \mid X=x,Z=z)\right)\mathclose{} \cdot\mathbb{P}(X=x \mid Z=z) && \text{(definition of } {\mathbb{E}\mathopen{}\left[Y \mid X=x,Z=z\right]\mathclose{}} \text{)} \\&= \sum_{y \in \mathcal{R}(Y)} y \cdot\sum_{x \in \mathcal{R}(X)} \mathbb{P}(Y=y \mid X=x,Z=z) \cdot\mathbb{P}(X=x \mid Z=z) && \text{(exchange order of summation)} \\&= \sum_{y \in \mathcal{R}(Y)} y \cdot\mathbb{P}(Y=y \mid Z=z) && \text{(law of total probability given } Z=z \text{)} \\&= {\mathbb{E}\mathopen{}\left[Y \mid Z=z\right]\mathclose{}} && \text{(definition of conditional expectation given } Z=z \text{)} \end{aligned} \]

Continuous case.

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y \mid X,Z\right]\mathclose{}} \mid Z=z\right]\mathclose{}} &= \int_{x \in \mathcal{R}(X)} {\mathbb{E}\mathopen{}\left[Y \mid X=x,Z=z\right]\mathclose{}} \cdot\mathbb{p}(X=x \mid Z=z)\, dx && \text{(expectation given } Z=z \text{)} \\&= \int_{x \in \mathcal{R}(X)} \mathopen{}\left(\int_{y \in \mathcal{R}(Y)} y \cdot\mathbb{p}(Y=y \mid X=x,Z=z)\, dy\right)\mathclose{} \cdot\mathbb{p}(X=x \mid Z=z)\, dx && \text{(definition of } {\mathbb{E}\mathopen{}\left[Y \mid X=x,Z=z\right]\mathclose{}} \text{)} \\&= \int_{x \in \mathcal{R}(X)} \int_{y \in \mathcal{R}(Y)} y \cdot\mathbb{p}(X=x, Y=y \mid Z=z)\, dy\, dx && \text{(product of conditional densities is the joint conditional density)} \\&= \int_{y \in \mathcal{R}(Y)} y \cdot\mathopen{}\left(\int_{x \in \mathcal{R}(X)} \mathbb{p}(X=x, Y=y \mid Z=z)\, dx\right)\mathclose{}\, dy && \text{(Fubini's theorem)} \\&= \int_{y \in \mathcal{R}(Y)} y \cdot\mathbb{p}(Y=y \mid Z=z)\, dy && \text{(marginalize over } x \text{)} \\&= {\mathbb{E}\mathopen{}\left[Y \mid Z=z\right]\mathclose{}} && \text{(definition of conditional expectation given } Z=z \text{)} \end{aligned} \]

Since the two sides agree at every such \(z\), they agree as random variables (functions of \(Z\)): \({\mathbb{E}\mathopen{}\left[Y \mid Z\right]\mathclose{}} = {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y \mid X,Z\right]\mathclose{}} \mid Z\right]\mathclose{}}\).

Remark. This identity is the tower rule applied conditionally on \(Z\).

Example 22 (Marginal expectation from conditional expectations) Suppose \(X\) is a binary random variable indicating treatment assignment (\(X=1\) treated, \(X=0\) control), with \(\mathbb{P}(X=1) = 0.5\), and suppose the outcome \(Y\) has conditional expectations:

\[{\mathbb{E}\mathopen{}\left[Y \mid X=1\right]\mathclose{}} = 10, \quad {\mathbb{E}\mathopen{}\left[Y \mid X=0\right]\mathclose{}} = 6\]

By the law of iterated expectations (Theorem 10):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y \mid X\right]\mathclose{}}\right]\mathclose{}} && \text{(law of iterated expectations)} \\&= {\mathbb{E}\mathopen{}\left[Y \mid X=1\right]\mathclose{}} \cdot\mathbb{P}(X=1) + {\mathbb{E}\mathopen{}\left[Y \mid X=0\right]\mathclose{}} \cdot\mathbb{P}(X=0) && \text{(LOTUS over the two values of } X \text{)} \\&= 10 \cdot 0.5 + 6 \cdot 0.5 && \text{(substitute)} \\&= 5 + 3 && \text{(multiply)} \\&= 8 && \text{(add)} \end{aligned} \]

Exercise 6 (Both-discrete case, infinite support: joint PMF) Let \(X\) and \(Y\) be independent, each Geometric on

\[ \begin{aligned} \mathcal{R}(X) &= \mathcal{R}(Y) \\ &= \{0, 1, 2, \dots\} \end{aligned} \]

with \(\mathbb{P}(X = x) = (1-p)\,p^x\) for a fixed \(p \in (0, 1)\) (\(X\) counts the number of failures before the first success in a sequence of independent trials with success probability \(1-p\); likewise for \(Y\)). Unlike Exercise 5, the support here is countably infinite. The joint PMF is \(\mathbb{P}(X = x,\, Y = y) = (1-p)^2\,p^{x+y}\).

Compute \({\mathbb{E}\mathopen{}\left[X + Y\right]\mathclose{}}\).

Solution. Compute \({\mathbb{E}\mathopen{}\left[X + Y\right]\mathclose{}}\) using Corollary 3 with

\[ \begin{aligned} \mu_X &= \mu_Y \\ &= \text{counting measure} \end{aligned} \]

and \(h(x, y) = x + y\). Since \(h(x,y) = x + y \ge 0\) for every \((x,y)\) in this support, condition (a) holds, so Corollary 3 (via Tonelli’s theorem) guarantees the order of this now-infinite double sum is exchangeable — unlike the finite case, elementary algebra alone could not establish this.

The derivation uses the standard geometric-series facts \(\sum_{y=0}^{\infty} p^y = \frac{1}{1-p}\) and \(\sum_{y=0}^{\infty} y\,p^y = \frac{p}{(1-p)^2}\) (e.g. Casella and Berger [6]). By Corollary 3 (both-discrete case), summing over \(y\) first for each fixed \(x\):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X + Y\right]\mathclose{}} &= \sum_{x=0}^{\infty} \sum_{y=0}^{\infty} (x + y)\,\mathbb{P}(X = x,\, Y = y) && \text{(joint-distribution form of Fubini--Tonelli)} \\&= \sum_{x=0}^{\infty} \sum_{y=0}^{\infty} (x + y)(1-p)^2 p^{x+y} && \text{(substitute the joint PMF)} \\&= \sum_{x=0}^{\infty} (1-p)^2 p^x \sum_{y=0}^{\infty} (x + y)\,p^y && \text{(factor } (1-p)^2 p^x \text{ out of the inner sum)} \\&= \sum_{x=0}^{\infty} (1-p)^2 p^x \mathopen{}\left(x \sum_{y=0}^{\infty} p^y + \sum_{y=0}^{\infty} y\,p^y\right)\mathclose{} && \text{(split the inner sum; factor out } x \text{)} \\&= \sum_{x=0}^{\infty} (1-p)^2 p^x \mathopen{}\left(\frac{x}{1-p} + \frac{p}{(1-p)^2}\right)\mathclose{} && \text{(geometric-series facts)} \\&= \sum_{x=0}^{\infty} p^x \mathopen{}\left[x(1-p) + p\right]\mathclose{} && \text{(multiply } (1-p)^2 \text{ into the parentheses)} \\&= (1-p) \sum_{x=0}^{\infty} x\,p^x + p \sum_{x=0}^{\infty} p^x && \text{(distribute the sum over the terms; factor out the constants)} \\&= (1-p) \cdot\frac{p}{(1-p)^2} + p \cdot\frac{1}{1-p} && \text{(geometric-series facts)} \\&= \frac{p}{1-p} + \frac{p}{1-p} && \text{(cancel } 1 - p \text{)} \\&= \frac{2p}{1-p} && \text{(add)} \end{aligned} \]

As a check,

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} \\ &= \frac{p}{1-p} \end{aligned} \]

(the mean of this Geometric distribution; Casella and Berger [6]), so

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X + Y\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} + {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} \\ &= \frac{2p}{1-p} \end{aligned} \]

by linearity of expectation, matching.

p <- 0.4
exact_sum <- 2 * p / (1 - p)

n_trunc <- 500
support <- 0:n_trunc
joint_pmf_fn <- function(x, y) (1 - p)^2 * p^(x + y)
joint_probs <- outer(support, support, joint_pmf_fn)
h_vals <- outer(support, support, function(x, y) x + y)

sum_y_first <- sum(rowSums(h_vals * joint_probs))
sum_x_first <- sum(colSums(h_vals * joint_probs))

tibble::tibble(
  quantity = c("exact (closed form)", "sum over y first, then x",
               "sum over x first, then y"),
  value = round(c(exact_sum, sum_y_first, sum_x_first), 6)
) |>
  pander::pander()

quantity	value
exact (closed form)	1.333
sum over y first, then x	1.333
sum over x first, then y	1.333

With \(p = 0.4\), the truncated sums agree with the closed form \(\frac{2p}{1-p} = 1.3333\) up to truncation error, which confirms the closed form. They cannot illustrate Tonelli’s guarantee, though: each truncation is a finite sum, and a finite sum gives the same total in either order, whether or not the infinite sums would agree. That the infinite sums agree is what Tonelli’s theorem supplies.

Tonelli’s nonnegativity condition, and what happens without it

The calculation in Exercise 6 only needed condition (a), \(h(X,Y) \ge 0\), because \(h(x,y) = x+y\) is nonnegative on this support. For a signed \(h\), interchanging an infinite double sum is not automatically valid — Corollary 3’s condition (b), \({\mathbb{E}\mathopen{}\left[\mathopen{}\left|h(X,Y)\right|\mathclose{}\right]\mathclose{}} < \infty\), is what licenses it in that case. Without either condition, the two orders can genuinely disagree. A standard example (a signed array, not a probability distribution; see e.g. [11], Theorem 3.54, p. 76 for the general theory of rearranging series): let \(a_{m,n} = 1\) if \(m = n\), \(a_{m,n} = -1\) if \(m = n+1\), and \(a_{m,n} = 0\) otherwise, for \(m, n = 0, 1, 2, \dots\). Summing each row \(m\) first: row \(0\) has only the term \(a_{0,0}=1\) (there is no valid \(n = -1\)), so its row sum is \(1\); every row \(m \ge 1\) has \(a_{m,m} = 1\) and \(a_{m,m-1} = -1\), so its row sum is \(0\). Summing the rows then gives \(1 + 0 + 0 + \cdots = 1\). Summing each column \(n\) first: every column \(n \ge 0\) has \(a_{n,n} = 1\) and \(a_{n+1,n} = -1\), so its column sum is always \(0\), and summing the columns then gives \(0 + 0 + \cdots = 0\). The two orders give \(1\) and \(0\): genuinely different answers, confirming that a condition like (a) or (b) really is needed once the terms are no longer all nonnegative.

Exercise 7 (Mixed case: one continuous variable, one discrete variable) Let \(Y \sim \mathrm{Bernoulli}(0.6)\) and, given \(Y = y\), let \(X \mid Y = y \sim \mathrm{Uniform}(0,\, y + 1)\).

Compute \({\mathbb{E}\mathopen{}\left[X\right]\mathclose{}}\).

Solution. Compute \({\mathbb{E}\mathopen{}\left[X\right]\mathclose{}}\) using Corollary 3 with \(\mu_X = \text{Lebesgue measure}\), \(\mu_Y = \text{counting measure}\), and \(h(x, y) = x\).

The joint density w.r.t. Lebesgue \(\times\) counting measure is \(f_{X,Y}(x, y) = f_{X \mid Y}(x \mid y)\,\mathbb{P}(Y = y)\):

\[ \begin{aligned} f_{X,Y}(x,\, 0) &= 1 \cdot 0.4 \\ &= 0.4 &&\text{ for } x \in [0,1]; \\ f_{X,Y}(x,\, 1) &= \tfrac{1}{2} \cdot 0.6 \\ &= 0.3 &&\text{ for } x \in [0,2]. \end{aligned} \]

By Corollary 3 (mixed case):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} &= \sum_{y \in \{0,1\}} \int_0^{y+1} x\,f_{X,Y}(x,\, y)\,dx && \text{(joint-distribution form of Fubini--Tonelli, mixed case)} \\ &= \int_0^1 x \cdot 0.4\,dx + \int_0^2 x \cdot 0.3\,dx && \text{(expand the sum over } y \in \{0, 1\} \text{; substitute } f_{X,Y} \text{)} \\ &= 0.4 \cdot \frac{1}{2} + 0.3 \cdot 2 && \text{(} \textstyle\int_0^b x\,dx = b^2/2 \text{)} \\ &= 0.2 + 0.6 && \text{(multiply)} \\ &= 0.8 && \text{(add)} \end{aligned} \]

As a check using the law of iterated expectations (Theorem 10): \({\mathbb{E}\mathopen{}\left[X \mid Y = 0\right]\mathclose{}} = \tfrac{1}{2}\) and \({\mathbb{E}\mathopen{}\left[X \mid Y = 1\right]\mathclose{}} = 1\), so

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} &= \tfrac{1}{2}(0.4) + 1(0.6) \\ &= 0.2 + 0.6 \\ &= 0.8. \end{aligned} \]

x_fine <- seq(0, 2, by = 0.005)
df <- tibble::tibble(
  x = c(x_fine[x_fine <= 1], x_fine),
  density = c(rep(0.4, sum(x_fine <= 1)), rep(0.3, length(x_fine))),
  label = c(
    rep("Y = 0  (P = 0.4)", sum(x_fine <= 1)),
    rep("Y = 1  (P = 0.6)", length(x_fine))
  )
)

plotly::plot_ly(
  df, x = ~x, y = ~density, color = ~label,
  colors = c("steelblue", "tomato")
) |>
  plotly::add_lines() |>
  plotly::layout(
    xaxis = list(title = "Value of X"),
    yaxis = list(
      title = "Joint density, f<sub>X,Y</sub>(x, y)",
      range = c(0, 0.55)
    ),
    legend = list(title = list(text = "Y value"))
  )

Figure 4: Joint density \(f_{X,Y}(x, y) = f_{X \mid Y}(x \mid y)\,\mathbb{P}(Y = y)\) for each value of the discrete variable \(Y\). The area under each component integrates to \(\mathbb{P}(Y = y)\): \(0.4 \cdot 1 = 0.4\) (blue) and \(0.3 \cdot 2 = 0.6\) (red), summing to 1.

Remark. \(Y\) takes only finitely many values here (two), so \(\sum_{y \in \{0,1\}} \int_0^{y+1} x\,f_{X,Y}(x,\,y)\,dx\) is just linearity of the integral applied to a two-term sum — \(\int g + \int k = \int (g + k)\) — not a genuine interchange of summation and integration order. If \(Y\) had a countably infinite range instead, the sum of integrals would be an infinite series, and Corollary 3’s guarantee would be load-bearing.

Exercise 8 (Mixed case, infinite discrete support) Let \(Y\) be Geometric on \(\{0, 1, 2, \dots\}\) with \(\mathbb{P}(Y = y) = (1-q)\,q^y\) for a fixed \(q \in (0, 1)\) and, given \(Y = y\), let \(X \mid Y = y \sim \mathrm{Uniform}(0,\, y+1)\). Unlike Exercise 7, \(Y\)’s range here is countably infinite.

Compute \({\mathbb{E}\mathopen{}\left[X\right]\mathclose{}}\).

Solution. Compute \({\mathbb{E}\mathopen{}\left[X\right]\mathclose{}}\) using Corollary 3 with \(\mu_X = \text{Lebesgue measure}\), \(\mu_Y = \text{counting measure}\), and \(h(x, y) = x\).

The joint density w.r.t. Lebesgue \(\times\) counting measure is

\[ \begin{aligned} f_{X,Y}(x, y) &= f_{X \mid Y}(x \mid y)\,\mathbb{P}(Y = y) \\ &= \frac{(1-q)\,q^y}{y+1} \end{aligned} \]

for \(x \in [0, y+1]\).

Since \(h(x, y) = x \ge 0\) on this support, condition (a) holds, so Corollary 3 (via Tonelli’s theorem) guarantees the now-infinite sum-of-integrals expression is valid — unlike in Exercise 7, this Fubini–Tonelli justification is required because the sum is infinite rather than finite.

By Corollary 3 (mixed case):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} &= \sum_{y=0}^{\infty} \int_0^{y+1} x\,f_{X,Y}(x,\,y)\,dx && \text{(joint-distribution form of Fubini--Tonelli, mixed case)} \\&= \sum_{y=0}^{\infty} \frac{(1-q)\,q^y}{y+1} \int_0^{y+1} x\,dx && \text{(} f_{X,Y}(x, y) \text{ is constant in } x \text{ on } [0, y+1] \text{)} \\&= \sum_{y=0}^{\infty} \frac{(1-q)\,q^y}{y+1} \cdot\frac{(y+1)^2}{2} && \text{(integrate)} \\&= \frac{1-q}{2} \sum_{y=0}^{\infty} (y+1)\,q^y && \text{(cancel } y + 1 \text{; factor out } \tfrac{1-q}{2} \text{)} \\&= \frac{1-q}{2} \mathopen{}\left(\sum_{y=0}^{\infty} y\,q^y + \sum_{y=0}^{\infty} q^y\right)\mathclose{} && \text{(distribute the sum over the terms)} \\&= \frac{1-q}{2} \mathopen{}\left(\frac{q}{(1-q)^2} + \frac{1}{1-q}\right)\mathclose{} && \text{(geometric-series facts)} \\&= \frac{1-q}{2} \cdot\frac{q + (1-q)}{(1-q)^2} && \text{(common denominator)} \\&= \frac{1-q}{2} \cdot\frac{1}{(1-q)^2} && \text{(simplify the numerator)} \\&= \frac{1}{2(1-q)} && \text{(cancel } 1 - q \text{)} \end{aligned} \]

using the same geometric-series facts as Exercise 6 (e.g. Casella and Berger [6]).

As a check using the law of iterated expectations (Theorem 10) and the expectation of a linear function (Corollary 1): \({\mathbb{E}\mathopen{}\left[X \mid Y=y\right]\mathclose{}} = \frac{y+1}{2}\) (the mean of \(\mathrm{Uniform}(0,y+1)\)) and \({\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} = \frac{q}{1-q}\) (the mean of this Geometric distribution; Casella and Berger [6]), so:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[X \mid Y\right]\mathclose{}}\right]\mathclose{}} && \text{(law of iterated expectations)} \\&= {\mathbb{E}\mathopen{}\left[\frac{Y+1}{2}\right]\mathclose{}} && \text{(} {\mathbb{E}\mathopen{}\left[X \mid Y = y\right]\mathclose{}} = \tfrac{y+1}{2} \text{)} \\&= \frac{{\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} + 1}{2} && \text{(expectation of a linear function of one random variable)} \\&= \frac{1}{2}\mathopen{}\left(\frac{q}{1-q} + 1\right)\mathclose{} && \text{(substitute } {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} \text{)} \\&= \frac{1}{2} \cdot\frac{q + (1-q)}{1-q} && \text{(common denominator)} \\&= \frac{1}{2(1-q)} && \text{(simplify the numerator)} \end{aligned} \]

matching.

q <- 0.5
exact_ex <- 1 / (2 * (1 - q))

n_trunc <- 2000
y_vals <- 0:n_trunc
py_vals <- (1 - q) * q^y_vals
cond_mean_x <- (y_vals + 1) / 2
trunc_ex <- sum(cond_mean_x * py_vals)

tibble::tibble(
  quantity = c("exact (closed form)", "truncated sum-of-integrals"),
  value = round(c(exact_ex, trunc_ex), 6)
) |>
  pander::pander()

quantity	value
exact (closed form)	1
truncated sum-of-integrals	1

With \(q = 0.5\), the truncated sum-of-integrals matches the closed form \(\frac{1}{2(1-q)} = 1\).

3 Loss and risk

Definition 9 (Prediction) A prediction of a random variable \(Y\) is a constant \(\hat y\), or a function \(\hat{Y} \stackrel{\text{def}}{=}g(X)\) of another observable random variable \(X\), used as a guess for the value \(Y\) takes:

\[\hat y\quad \text{or} \quad \hat{Y} \stackrel{\text{def}}{=}g(X)\]

Example 23 (Predictions of a die roll) Let \(Y\) be the number shown by a fair six-sided die. The constant \(\hat y= 3\) is a prediction of \(Y\).

Let \(X\) be the indicator that \(Y\) is even. The function \(g(X) = 2 + 2X\) is also a prediction of \(Y\): it guesses \(2\) when \(Y\) is odd and \(4\) when \(Y\) is even.

Remark 2 (A prediction from a random input is random). When no covariate or feature is available, a prediction is a single number \(\hat y\) (such as a constant \(c\)). When an informative random variable \(X\) is observed, a prediction is a function \(g(X)\), which is itself a random variable because \(X\) is random.

Definition 10 (Loss function) A loss function \(L(y, \hat y)\) is a rule that gives a number that is 0 or larger: the cost of predicting \(\hat y\) when the true value is \(y\) [12].

Example 24 (A loss function that charges more for overpredicting) Let \(L(y, \hat y)\) cost \(2\) per unit of overprediction and \(1\) per unit of underprediction:

\[ L(y, \hat y) = \begin{cases} 2\mathopen{}\left(\hat y- y\right)\mathclose{} & \text{if } \hat y> y \\ y - \hat y& \text{if } \hat y\le y. \end{cases} \]

This is a loss function because it is never negative. For the true value \(y = 2\) and the prediction \(\hat y= 7\), the loss is \(L(2, 7) = 2 \cdot 5 = 10\). For \(y = 7\) and \(\hat y= 2\), the loss is \(L(7, 2) = 5\).

Definition 11 (Squared error loss) The squared error loss is the loss function that charges the square of the difference between the true value \(y\) and the prediction \(\hat y\):

\[L(y, \hat y) \stackrel{\text{def}}{=}{\mathopen{}\left(y - \hat y\right)\mathclose{}}^{2}\]

Example 25 (Squared error loss for one prediction) For the true value \(y = 2\) and the prediction \(\hat y= 7\), the squared error loss is

\[{\mathopen{}\left(2 - 7\right)\mathclose{}}^{2} = 25.\]

Definition 12 (Absolute error loss) The absolute error loss is the loss function that charges the absolute value of the difference between the true value \(y\) and the prediction \(\hat y\):

\[L(y, \hat y) \stackrel{\text{def}}{=}\mathopen{}\left|y - \hat y\right|\mathclose{}\]

Example 26 (Absolute error loss for one prediction) For the true value \(y = 2\) and the prediction \(\hat y= 7\), the absolute error loss is

\[\mathopen{}\left|2 - 7\right|\mathclose{} = 5.\]

Remark 3 (Squared error loss is the most common choice). Hastie et al. [13, p. 18] call squared error loss “by far the most common and convenient” choice.

Definition 13 (Risk (expected loss)) For a random variable \(Y\) and a prediction \(\hat y\) (a constant, or a function \(g(X)\) of a random input \(X\)), the risk is the expectation of the loss [14]:

\[R \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[L(Y, \hat y)\right]\mathclose{}}\]

Example 27 (Risk of a constant prediction) Let \(Y\) be a random variable with \(\mathbb{P}(Y = 0) = 0.5\) and \(\mathbb{P}(Y = 2) = 0.5\), and use the constant prediction \(\hat y= 0\). The risk under squared error loss is

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y - 0\right)\mathclose{}}^{2}\right]\mathclose{}} &= 0.5 \cdot{\mathopen{}\left(0 - 0\right)\mathclose{}}^{2} + 0.5 \cdot{\mathopen{}\left(2 - 0\right)\mathclose{}}^{2} \\ &= 2. \end{aligned} \]

The risk under absolute error loss is

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[\mathopen{}\left|Y - 0\right|\mathclose{}\right]\mathclose{}} &= 0.5 \cdot\mathopen{}\left|0 - 0\right|\mathclose{} + 0.5 \cdot\mathopen{}\left|2 - 0\right|\mathclose{} \\ &= 1. \end{aligned} \]

Remark 4 (Risk measures the quality of a prediction). The quality of a prediction is evaluated using a loss function and its expected value, the risk.

Exercise 9 (Squared and absolute loss)  

1. For a true value \(y = 3\) and a prediction \(\hat y= 5\), compute the squared error loss \({\mathopen{}\left(y - \hat y\right)\mathclose{}}^{2}\) and the absolute error loss \(\mathopen{}\left|y - \hat y\right|\mathclose{}\).

2. A random variable \(Y\) has \(\mathbb{P}(Y = 0) = 0.5\) and \(\mathbb{P}(Y = 4) = 0.5\). For the constant prediction \(c = 1\), compute the risk under squared error loss \({\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y - c\right)\mathclose{}}^{2}\right]\mathclose{}}\) and the risk under absolute error loss \({\mathbb{E}\mathopen{}\left[\mathopen{}\left|Y - c\right|\mathclose{}\right]\mathclose{}}\).

3. Repeat part 2 for \(c = 2\).

4. Which of \(c = 1\) and \(c = 2\) gives the smaller risk under each loss?

Solution 5.

1. For squared error loss: \[{\mathopen{}\left(3 - 5\right)\mathclose{}}^{2} = 4\] For absolute error loss: \[\mathopen{}\left|3 - 5\right|\mathclose{} = 2\]

2. For \(c = 1\): The risk under squared error loss is: \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y - 1\right)\mathclose{}}^{2}\right]\mathclose{}} &= 0.5 \cdot{\mathopen{}\left(0 - 1\right)\mathclose{}}^{2} + 0.5 \cdot{\mathopen{}\left(4 - 1\right)\mathclose{}}^{2} \\ &= 0.5 \cdot 1 + 0.5 \cdot 9 \\ &= 5 \end{aligned} \] The risk under absolute error loss is: \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[\mathopen{}\left|Y - 1\right|\mathclose{}\right]\mathclose{}} &= 0.5 \cdot\mathopen{}\left|0 - 1\right|\mathclose{} + 0.5 \cdot\mathopen{}\left|4 - 1\right|\mathclose{} \\ &= 0.5 \cdot 1 + 0.5 \cdot 3 \\ &= 2 \end{aligned} \]

3. For \(c = 2\): The risk under squared error loss is: \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y - 2\right)\mathclose{}}^{2}\right]\mathclose{}} &= 0.5 \cdot{\mathopen{}\left(0 - 2\right)\mathclose{}}^{2} + 0.5 \cdot{\mathopen{}\left(4 - 2\right)\mathclose{}}^{2} \\ &= 0.5 \cdot 4 + 0.5 \cdot 4 \\ &= 4 \end{aligned} \] The risk under absolute error loss is: \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[\mathopen{}\left|Y - 2\right|\mathclose{}\right]\mathclose{}} &= 0.5 \cdot\mathopen{}\left|0 - 2\right|\mathclose{} + 0.5 \cdot\mathopen{}\left|4 - 2\right|\mathclose{} \\ &= 0.5 \cdot 2 + 0.5 \cdot 2 \\ &= 2 \end{aligned} \]

4. The prediction \(c = 2\) has the smaller risk under squared error loss (\(4\) versus \(5\)). The two predictions have the same risk under absolute error loss (\(2\)), so under absolute error loss this exercise does not choose one of them. The prediction \(c = 2\) equals the expectation: \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} &= 0.5 \cdot 0 + 0.5 \cdot 4 \\ &= 2 \end{aligned} \]

Remark 5 (The best prediction depends on the loss). In Exercise 9 (parts 2 and 3), the risk of the prediction \(c = 1\) is \(5\) for squared error loss and \(2\) for absolute error loss. The risk of the prediction \(c = 2\) is \(4\) for squared error loss and \(2\) for absolute error loss.

The prediction with the smallest risk depends on the loss. In Exercise 9, every constant \(c\) from \(0\) to \(4\) has absolute error risk \(2\), because

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[\mathopen{}\left|Y - c\right|\mathclose{}\right]\mathclose{}} &= 0.5\,c + 0.5\,(4 - c) \\ &= 2. \end{aligned} \]

Only \(c = 2\) has the smallest squared error risk. Under squared error loss, the prediction function that minimizes risk is the conditional mean \({\mathbb{E}\mathopen{}\left[Y \mid X\right]\mathclose{}}\), proved below (Theorem 13). Hastie et al. [13, p. 20] state that, for absolute error loss, the best prediction function is the conditional median instead of the conditional mean. This statement is given here without proof.

Exercise 10 (Comparing prediction functions under squared-error loss) Let \((X, Y)\) be jointly distributed discrete random variables. \(X\) takes values in \(\{0, 1\}\) with \(\mathbb{P}(X= 0) = 0.5\) and \(\mathbb{P}(X= 1) = 0.5\). Given \(X= 0\), \(Y\) takes values in \(\{0, 2\}\) with conditional probabilities \(\mathbb{P}(Y= 0 \mid X= 0) = 0.5\) and \(\mathbb{P}(Y= 2 \mid X= 0) = 0.5\). Given \(X= 1\), \(Y\) takes values in \(\{4, 6\}\) with conditional probabilities \(\mathbb{P}(Y= 4 \mid X= 1) = 0.5\) and \(\mathbb{P}(Y= 6 \mid X= 1) = 0.5\).

1. Compute the marginal mean \({\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}}\) and the conditional expectation function \({f^*\mathopen{}\left(X\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\).

2. Consider the constant prediction \(f_1\mathopen{}\left(X\right)\mathclose{} \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} = 3\), which ignores \(X\). Compute its squared-error risk \({\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- f_1\mathopen{}\left(X\right)\mathclose{}\right)\mathclose{}}^{2}\right]\mathclose{}}\).

3. Consider the alternate prediction \(f_2\mathopen{}\left(X\right)\mathclose{} \stackrel{\text{def}}{=}4X\). Compute its squared-error risk \({\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- f_2\mathopen{}\left(X\right)\mathclose{}\right)\mathclose{}}^{2}\right]\mathclose{}}\).

4. Compute the squared-error risk \({\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- {f^*\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}}\) of the conditional mean predictor \({f^*\mathopen{}\left(X\right)\mathclose{}}\). Which of \(f_1\), \(f_2\), and \(f^*\) achieves the lowest risk?

Solution 6.

1. For each value of \(X\), compute the conditional expectation: \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y\mid X= 0\right]\mathclose{}} &= 0 \cdot 0.5 + 2 \cdot 0.5 \\ &= 1 \end{aligned} \] \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y\mid X= 1\right]\mathclose{}} &= 4 \cdot 0.5 + 6 \cdot 0.5 \\ &= 5 \end{aligned} \] By the law of iterated expectations (Theorem 10), the marginal mean is: \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\right]\mathclose{}} \\ &= 1 \cdot 0.5 + 5 \cdot 0.5 \\ &= 3 \end{aligned} \] The conditional expectation function is \({f^*\mathopen{}\left(0\right)\mathclose{}} = 1\) and \({f^*\mathopen{}\left(1\right)\mathclose{}} = 5\), which can also be written \({f^*\mathopen{}\left(X\right)\mathclose{}} = 1 + 4X\).

2. For the constant prediction \(f_1\mathopen{}\left(X\right)\mathclose{} = 3\): Given \(X= 0\): \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- 3\right)\mathclose{}}^{2} \mid X= 0\right]\mathclose{}} &= {\mathopen{}\left(0 - 3\right)\mathclose{}}^{2} \cdot 0.5 + {\mathopen{}\left(2 - 3\right)\mathclose{}}^{2} \cdot 0.5 \\ &= 9 \cdot 0.5 + 1 \cdot 0.5 \\ &= 5 \end{aligned} \] Given \(X= 1\): \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- 3\right)\mathclose{}}^{2} \mid X= 1\right]\mathclose{}} &= {\mathopen{}\left(4 - 3\right)\mathclose{}}^{2} \cdot 0.5 + {\mathopen{}\left(6 - 3\right)\mathclose{}}^{2} \cdot 0.5 \\ &= 1 \cdot 0.5 + 9 \cdot 0.5 \\ &= 5 \end{aligned} \] By the law of iterated expectations (Theorem 10): \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- f_1\mathopen{}\left(X\right)\mathclose{}\right)\mathclose{}}^{2}\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- 3\right)\mathclose{}}^{2} \mid X\right]\mathclose{}}\right]\mathclose{}} \\ &= 5 \cdot 0.5 + 5 \cdot 0.5 \\ &= 5 \end{aligned} \]

3. For the prediction \(f_2\mathopen{}\left(X\right)\mathclose{} = 4X\): Here \(f_2\mathopen{}\left(0\right)\mathclose{} = 0\) and \(f_2\mathopen{}\left(1\right)\mathclose{} = 4\). Given \(X= 0\): \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- 0\right)\mathclose{}}^{2} \mid X= 0\right]\mathclose{}} &= {\mathopen{}\left(0 - 0\right)\mathclose{}}^{2} \cdot 0.5 + {\mathopen{}\left(2 - 0\right)\mathclose{}}^{2} \cdot 0.5 \\ &= 0 \cdot 0.5 + 4 \cdot 0.5 \\ &= 2 \end{aligned} \] Given \(X= 1\): \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- 4\right)\mathclose{}}^{2} \mid X= 1\right]\mathclose{}} &= {\mathopen{}\left(4 - 4\right)\mathclose{}}^{2} \cdot 0.5 + {\mathopen{}\left(6 - 4\right)\mathclose{}}^{2} \cdot 0.5 \\ &= 0 \cdot 0.5 + 4 \cdot 0.5 \\ &= 2 \end{aligned} \] By Theorem 10: \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- f_2\mathopen{}\left(X\right)\mathclose{}\right)\mathclose{}}^{2}\right]\mathclose{}} &= 2 \cdot 0.5 + 2 \cdot 0.5 \\ &= 2 \end{aligned} \]

4. For the conditional mean predictor \({f^*\mathopen{}\left(X\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\): Given \(X= 0\), \({f^*\mathopen{}\left(0\right)\mathclose{}} = 1\): \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- 1\right)\mathclose{}}^{2} \mid X= 0\right]\mathclose{}} &= {\mathopen{}\left(0 - 1\right)\mathclose{}}^{2} \cdot 0.5 + {\mathopen{}\left(2 - 1\right)\mathclose{}}^{2} \cdot 0.5 \\ &= 1 \cdot 0.5 + 1 \cdot 0.5 \\ &= 1 \end{aligned} \] Given \(X= 1\), \({f^*\mathopen{}\left(1\right)\mathclose{}} = 5\): \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- 5\right)\mathclose{}}^{2} \mid X= 1\right]\mathclose{}} &= {\mathopen{}\left(4 - 5\right)\mathclose{}}^{2} \cdot 0.5 + {\mathopen{}\left(6 - 5\right)\mathclose{}}^{2} \cdot 0.5 \\ &= 1 \cdot 0.5 + 1 \cdot 0.5 \\ &= 1 \end{aligned} \] By Theorem 10: \[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- {f^*\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} &= 1 \cdot 0.5 + 1 \cdot 0.5 \\ &= 1 \end{aligned} \] The conditional mean predictor \(f^*\) achieves the lowest risk (\(1\)), strictly outperforming \(f_2\) (\(2\)) and the constant prediction \(f_1\) (\(5\)).

Definition 14 (Outcome) The outcome \(Y\) of a data point is the random variable whose value a prediction tries to guess.

Example 28 (The outcome in the best-predictor exercise) In Exercise 10, the outcome \(Y\) takes the values \(0\) and \(2\) when the input is \(0\), and the values \(4\) and \(6\) when the input is \(1\).

Definition 15 (Input) The input \(X\) of a data point (also called its predictor or covariate) is the observed random variable from which a prediction of its outcome is computed.

Example 29 (The input in the best-predictor exercise) In Exercise 10, the input \(X\) takes the values \(0\) and \(1\), each with probability \(0.5\).

Definition 16 (Prediction function) A prediction function \(f\) is a function that takes a value \(x\) of the input and returns a prediction \({f\mathopen{}\left(x\right)\mathclose{}}\) of the outcome.

Example 30 (A constant prediction function) In Exercise 10, the function \(f_1\mathopen{}\left(x\right)\mathclose{} = 3\) is a prediction function: it returns the prediction \(3\) for every value of the input.

Definition 17 (New observation) A new observation is an input and outcome pair \((X, Y)\) with the same joint distribution as the observations used to choose the prediction, statistically independent of them, whose input is observed and whose outcome is not yet observed.

Example 31 (A new observation in the best-predictor exercise) In Exercise 10, suppose the new observation \((X, Y)\) satisfies the suppositions of its definition, with the joint distribution given in the exercise as the population it is drawn from. Suppose also that its input is observed to be \(X= 1\). Then its unobserved outcome is \(4\) or \(6\) with equal probability, and a prediction of \(5\) is a guess for that outcome.

Definition 18 (Prediction error) The prediction error of a prediction \(\hat{Y}\) of an outcome \(Y\) is the prediction minus the outcome:

\[{{\varepsilon_{\text{pred}}}\mathopen{}\left(\hat{Y}\right)\mathclose{}} \stackrel{\text{def}}{=}\hat{Y} - Y\]

Example 32 (Prediction error in the best-predictor exercise) In Exercise 10, consider the constant prediction \(f_1\mathopen{}\left(X\right)\mathclose{} = 3\). Its prediction error is \({{\varepsilon_{\text{pred}}}\mathopen{}\left(f_1\right)\mathclose{}} = 3 - Y\). This is \(3\) when \(Y= 0\). It is \(-3\) when \(Y= 6\).

Remark 6 (The prediction error of a prediction function). A prediction function \(f\) makes the prediction \({f\mathopen{}\left(X\right)\mathclose{}}\) for the outcome \(Y\) of a new observation with input \(X\). We write \({{\varepsilon_{\text{pred}}}\mathopen{}\left(f\right)\mathclose{}}\) for the prediction error of that prediction, meaning \({{\varepsilon_{\text{pred}}}\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}\). It is

\[{{\varepsilon_{\text{pred}}}\mathopen{}\left(f\right)\mathclose{}} = {f\mathopen{}\left(X\right)\mathclose{}} - Y\]

The reference value in a prediction error is the observed outcome \(Y\), not the conditional mean of that outcome.

Definition 19 (Squared-error risk (expected prediction error)) The squared-error risk of a prediction function \(f\), also called its expected prediction error [13, p. 18], is the expected square of its prediction error:

\[{R\mathopen{}\left(f\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[{\mathopen{}\left({{\varepsilon_{\text{pred}}}\mathopen{}\left(f\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}}\]

Example 33 (Squared-error risk in the best-predictor exercise) In Exercise 10, consider the constant prediction \(f_1\mathopen{}\left(X\right)\mathclose{} = 3\). Its squared-error risk is \({R\mathopen{}\left(f_1\right)\mathclose{}} = 5\).

Remark 7 (The squared-error risk is a risk). The squared-error risk is the risk under squared error loss.

Lemma 2 (Prediction error splits at the conditional mean) Let \({\mu\mathopen{}\left(X\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\) be the conditional expectation function of \(Y\) given \(X\). For any prediction function \(f\), the prediction error of \(f\) is the prediction error of \(\mu\) plus the difference \({\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}\) between the prediction \({f\mathopen{}\left(X\right)\mathclose{}}\) and \({\mu\mathopen{}\left(X\right)\mathclose{}}\):

\[{{\varepsilon_{\text{pred}}}\mathopen{}\left(f\right)\mathclose{}} = {{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}} + {\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}\]

Proof. The notation \({{\varepsilon_{\text{pred}}}\mathopen{}\left(f\right)\mathclose{}}\) is introduced in Remark 6.

\[ \begin{aligned} {{\varepsilon_{\text{pred}}}\mathopen{}\left(f\right)\mathclose{}} &= {f\mathopen{}\left(X\right)\mathclose{}} - Y && \text{(prediction error notation)} \\ &= \mathopen{}\left({\mu\mathopen{}\left(X\right)\mathclose{}} - Y\right)\mathclose{} + {\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}} && \text{(add and subtract } {\mu\mathopen{}\left(X\right)\mathclose{}} \text{)} \\ &= {{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}} + {\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}} && \text{(prediction error notation)} \end{aligned} \]

Example 34 (The split in the best-predictor exercise) In Exercise 10, when \(X= 0\) and \(Y= 0\), the constant prediction function \(f_1\), with \(f_1\mathopen{}\left(X\right)\mathclose{} = 3\), has prediction error \({{\varepsilon_{\text{pred}}}\mathopen{}\left(f_1\right)\mathclose{}} = 3\). The conditional mean is \({\mu\mathopen{}\left(0\right)\mathclose{}} = 1\), so \({{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}} = 1\) and \({\mathopen{}\left(f_1\mathopen{}\left(0\right)\mathclose{} - {\mu\mathopen{}\left(0\right)\mathclose{}}\right)\mathclose{}} = 2\), and \(3 = 1 + 2\).

Lemma 3 (The conditional mean’s error is uncorrelated with any function of the input) Let \({\mu\mathopen{}\left(X\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\) be the conditional expectation function of \(Y\) given \(X\). For any prediction function \(f\) with \({\mathbb{E}\mathopen{}\left[{f\mathopen{}\left(X\right)\mathclose{}}^2\right]\mathclose{}} < \infty\):

\[{\mathbb{E}\mathopen{}\left[{{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}} \cdot\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}\right]\mathclose{}} = 0\]

Proof. Because \({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\) is a function of \(X\), by Theorem 8 it factors out of the conditional expectation:

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}}\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{} \mid X\right]\mathclose{}} &= \mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{} \cdot{\mathbb{E}\mathopen{}\left[{{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}} \mid X\right]\mathclose{}} && \text{(a function of } X\text{ factors out)} \\ &= \mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{} \cdot{\mathbb{E}\mathopen{}\left[{\mu\mathopen{}\left(X\right)\mathclose{}} - Y\mid X\right]\mathclose{}} && \text{(definition of } {{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}} \text{)} \\ &= \mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{} \cdot\mathopen{}\left({\mathbb{E}\mathopen{}\left[{\mu\mathopen{}\left(X\right)\mathclose{}} \mid X\right]\mathclose{}} - {\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\right)\mathclose{} && \text{(linearity of conditional expectation)} \\ &= \mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{} \cdot\mathopen{}\left({\mu\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{} && \text{(a function of } X\text{ is its own conditional mean, and definition of } {\mu\mathopen{}\left(X\right)\mathclose{}} \text{)} \\ &= 0 && \text{(evaluate)} \end{aligned} \]

By the law of iterated expectations (Theorem 10):

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[{{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}}\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[{\mathbb{E}\mathopen{}\left[{{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}}\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{} \mid X\right]\mathclose{}}\right]\mathclose{}} && \text{(law of iterated expectations)} \\ &= {\mathbb{E}\mathopen{}\left[0\right]\mathclose{}} && \text{(the conditional expectation is } 0 \text{)} \\ &= 0 && \text{(expectation of a constant)} \end{aligned} \]

Example 35 (The vanishing cross term in the best-predictor exercise) In Exercise 10, take the constant prediction \(f_1\mathopen{}\left(X\right)\mathclose{} = 3\). When \(X= 0\), \({\mu\mathopen{}\left(0\right)\mathclose{}} = 1\), so \(f_1\mathopen{}\left(0\right)\mathclose{} - {\mu\mathopen{}\left(0\right)\mathclose{}} = 2\), and \({{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}} = 1 - Y\) is \(1\) or \(-1\) with equal probability, so the product has conditional mean \(0\). The same cancellation happens when \(X= 1\), so the expectation of the product is \(0\).

Theorem 12 (Squared-error risk decomposition) Let \({\mu\mathopen{}\left(X\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\) be the conditional expectation function of \(Y\) given \(X\). For any prediction function \(f\) with \({\mathbb{E}\mathopen{}\left[{f\mathopen{}\left(X\right)\mathclose{}}^2\right]\mathclose{}} < \infty\):

\[{R\mathopen{}\left(f\right)\mathclose{}} = {R\mathopen{}\left(\mu\right)\mathclose{}} + {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}}\]

Proof. The steps use the squared-error risk definition, Lemma 2, Lemma 3, and Theorem 4:

\[ \begin{aligned} {R\mathopen{}\left(f\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({{\varepsilon_{\text{pred}}}\mathopen{}\left(f\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} && \text{(definition of the squared-error risk)} \\ &= {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}} + {\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} && \text{(the prediction error splits at } \mu\text{)} \\ &= {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} + 2 \cdot{\mathbb{E}\mathopen{}\left[{{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}} \cdot{\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}\right]\mathclose{}} + {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} && \text{(expand the square; linearity of expectation)} \\ &= {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} + 2 \cdot 0 + {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} && \text{(the cross term is } 0 \text{)} \\ &= {R\mathopen{}\left(\mu\right)\mathclose{}} + {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} && \text{(definition of the squared-error risk of } \mu\text{)} \end{aligned} \]

Example 36 (The risk decomposition in the best-predictor exercise) In Exercise 10, the constant prediction \(f_1\mathopen{}\left(X\right)\mathclose{} = 3\) has squared-error risk \({R\mathopen{}\left(f_1\right)\mathclose{}} = 5\), which Theorem 12 splits as \(5 = 1 + 4\): the risk \(1\) of the conditional expectation function plus the excess \(4\) from using \(f_1\).

Theorem 13 (Conditional mean minimizes squared-error risk) Let \(X\) and \(Y\) be jointly distributed random variables with \({\mathbb{E}\mathopen{}\left[Y^2\right]\mathclose{}} < \infty\). Let \({\mu\mathopen{}\left(X\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\) be the conditional expectation function of \(Y\) given \(X\). Write \(f^*\stackrel{\text{def}}{=}\mu\) for the prediction function that uses it. Then for any prediction function \(f\) with \({\mathbb{E}\mathopen{}\left[{f\mathopen{}\left(X\right)\mathclose{}}^2\right]\mathclose{}} < \infty\):

\[{R\mathopen{}\left(f\right)\mathclose{}} \ge {R\mathopen{}\left(f^*\right)\mathclose{}}\]

with equality if and only if \(\mathbb{P}({f\mathopen{}\left(X\right)\mathclose{}} = {f^*\mathopen{}\left(X\right)\mathclose{}}) = 1\). That is, \(f^*\) minimizes the squared-error risk among all prediction functions of \(X\).

Proof. By Theorem 12:

\[ \begin{aligned} {R\mathopen{}\left(f\right)\mathclose{}} - {R\mathopen{}\left(f^*\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {f^*\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} && \text{(squared-error risk decomposition)} \\ &\ge 0 && \text{(the expectation of a non-negative variable)} \end{aligned} \]

The expectation of the non-negative variable \({\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {f^*\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}^{2}\) is zero if and only if the variable is \(0\) with probability \(1\), meaning \(\mathbb{P}({f\mathopen{}\left(X\right)\mathclose{}} = {f^*\mathopen{}\left(X\right)\mathclose{}}) = 1\).

Example 37 (The best predictor in the best-predictor exercise) In Exercise 10, consider the constant prediction \(f_1\mathopen{}\left(X\right)\mathclose{} = 3\). Its squared-error risk is \({R\mathopen{}\left(f_1\right)\mathclose{}} = 5\). The risk of the conditional mean is \({R\mathopen{}\left(f^*\right)\mathclose{}} = 1\). So \(f_1\) has the larger risk. The inequality is strict because \(f_1\mathopen{}\left(X\right)\mathclose{} \ne {f^*\mathopen{}\left(X\right)\mathclose{}}\).

Definition 20 (Residual risk (irreducible risk)) The residual risk (also called the irreducible risk) is the squared-error risk of the conditional expectation function \(\mu\) of Theorem 12:

\[{R\mathopen{}\left(\mu\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}}\]

Theorem 14 (The residual risk is the expected conditional variance) Let \(X\) and \(Y\) be as in Theorem 13. Then the residual risk is the expectation of the conditional variance of \(Y\) given \(X\):

\[ {R\mathopen{}\left(\mu\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[{\operatorname{Var}\mathopen{}\left(Y\mid X\right)\mathclose{}}\right]\mathclose{}}. \tag{1}\]

Proof. \[ \begin{aligned} {R\mathopen{}\left(\mu\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({{\varepsilon_{\text{pred}}}\mathopen{}\left(\mu\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} && \text{(definition of the residual risk)} \\ &= {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({\mu\mathopen{}\left(X\right)\mathclose{}} - Y\right)\mathclose{}}^{2}\right]\mathclose{}} && \text{(definition of the prediction error)} \\ &= {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} && \text{(a square is unchanged by negation)} \\ &= {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- {\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} && \text{(definition of } \mu\text{)} \\ &= {\mathbb{E}\mathopen{}\left[{{\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- {\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\right)\mathclose{}}^{2} \mid X\right]\mathclose{}}}\right]\mathclose{}} && \text{(law of total expectation)} \\ &= {\mathbb{E}\mathopen{}\left[{\operatorname{Var}\mathopen{}\left(Y\mid X\right)\mathclose{}}\right]\mathclose{}} && \text{(definition of conditional variance)} \end{aligned} \]

Remark 8 (The residual risk does not depend on the prediction function). The residual risk depends only on the joint distribution of \((X, Y)\) and not on the prediction function \(f\), so no choice of \(f\) can achieve a smaller risk. It is the minimum only among prediction functions of the given \(X\): the outcome may also depend on variables that \(X\) leaves out, and those variables are part of the unexplained variation \(Y- {\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\). Measuring one of them and adding it to the variables in \(X\) can lower the residual risk, as in Example 39. [15, Sec. 2.1.1], p. 18 makes the same point: the error term “may contain unmeasured variables that are useful in predicting [\(Y\)]”.

Example 38 (Residual risk in the best-predictor exercise) In Exercise 10, \({\operatorname{Var}\mathopen{}\left(Y\mid X= 0\right)\mathclose{}} = 1\) and \({\operatorname{Var}\mathopen{}\left(Y\mid X= 1\right)\mathclose{}} = 1\), so the residual risk is \({\mathbb{E}\mathopen{}\left[{\operatorname{Var}\mathopen{}\left(Y\mid X\right)\mathclose{}}\right]\mathclose{}} = 1\).

Example 39 (Adding a predictor lowers the residual risk) Let \(X_1\), \(X_2\) and \(\varepsilon\) be independent, with \({\operatorname{Var}\mathopen{}\left(X_2\right)\mathclose{}} = 1\) and \({\operatorname{Var}\mathopen{}\left(\varepsilon\right)\mathclose{}} = 1\), and let \(Y= X_1 + X_2 + \varepsilon\). A prediction from \(X_1\) alone has conditional variance \({\operatorname{Var}\mathopen{}\left(Y\mid X_1\right)\mathclose{}} = {\operatorname{Var}\mathopen{}\left(X_2\right)\mathclose{}} + {\operatorname{Var}\mathopen{}\left(\varepsilon\right)\mathclose{}} = 2\), so the residual risk is \(2\). A prediction from \((X_1, X_2)\) has conditional variance \({\operatorname{Var}\mathopen{}\left(Y\mid X_1, X_2\right)\mathclose{}} = {\operatorname{Var}\mathopen{}\left(\varepsilon\right)\mathclose{}} = 1\), so the residual risk is \(1\).

Definition 21 (Reducible risk) The reducible risk of a prediction function \(f\) is the expectation of the squared difference between \({f\mathopen{}\left(X\right)\mathclose{}}\) and the conditional mean \({\mu\mathopen{}\left(X\right)\mathclose{}}\) of the outcome:

\[{\mathbb{E}\mathopen{}\left[{\mathopen{}\left({f\mathopen{}\left(X\right)\mathclose{}} - {\mu\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}}\]

Remark 9 (When the reducible risk is zero). The reducible risk is zero if and only if the prediction function \(f\) equals the conditional expectation function with probability 1.

\[ {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}} - {f\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} = 0 \iff \mathbb{P}({f\mathopen{}\left(X\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}) = 1 \]

Example 40 (Reducible risk in the best-predictor exercise) In Exercise 10, the constant prediction function \(f_1\), with \(f_1\mathopen{}\left(X\right)\mathclose{} = 3\), has reducible risk \(5 - 1 = 4\), and the prediction function \(f_2\), with \(f_2\mathopen{}\left(X\right)\mathclose{} = 4X\), has reducible risk \(2 - 1 = 1\).

Remark. This characterization of the best predictor under squared error loss follows Hastie et al. [13, Sec. 2.4], eq. 2.11. The risk identity in Theorem 12

\[{\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- {f\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} = {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- {\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} + {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}} - {f\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}}\]

shows that the total squared-error risk is the sum of the residual risk (Definition 20) and the reducible risk (Definition 21). James et al. [15, Sec. 2.1.1], p. 18 calls the squared difference between a prediction function and the conditional mean the “reducible error”.

Corollary 4 (Conditional mean as the minimizer of squared-error risk) Let \(X\) and \(Y\) be jointly distributed random variables with \({\mathbb{E}\mathopen{}\left[Y^2\right]\mathclose{}} < \infty\). Among prediction functions \(f\) with \({\mathbb{E}\mathopen{}\left[{f\mathopen{}\left(X\right)\mathclose{}}^2\right]\mathclose{}} < \infty\), the conditional expectation function is a minimizer of the squared-error risk:

\[{\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}} \in \arg \min_{f} {\mathbb{E}\mathopen{}\left[{\mathopen{}\left(Y- {f\mathopen{}\left(X\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}},\]

where \(\arg \min\) is the set of minimizers.

Proof. By Theorem 13, \({f^*\mathopen{}\left(X\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}\) has risk no larger than that of any prediction function \(f\), so it is a minimizer.

Example 41 (The minimizer in the best-predictor exercise) In Exercise 10, the conditional expectation function \({f^*\mathopen{}\left(X\right)\mathclose{}}\) has risk \(1\). The constant prediction function \(f_1\), with \(f_1\mathopen{}\left(X\right)\mathclose{} = 3\), has risk \(5\) and \(f_2\mathopen{}\left(X\right)\mathclose{} = 4X\) has risk \(2\), both larger than \(1\). So \(f^*\) is the minimizer among the three.

Corollary 5 (The minimizer of squared-error risk is unique up to probability-1 equality) Let \(X\) and \(Y\) be jointly distributed random variables with \({\mathbb{E}\mathopen{}\left[Y^2\right]\mathclose{}} < \infty\). Every prediction function \(f\) with \({\mathbb{E}\mathopen{}\left[{f\mathopen{}\left(X\right)\mathclose{}}^2\right]\mathclose{}} < \infty\) that is a minimizer of the squared-error risk (see Corollary 4) satisfies:

\[\mathbb{P}({f\mathopen{}\left(X\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[Y\mid X\right]\mathclose{}}) = 1\]

Proof. If \(f\) is a minimizer, its risk equals that of \(f^*\), so the equality case of Theorem 13 gives \(\mathbb{P}({f\mathopen{}\left(X\right)\mathclose{}} = {f^*\mathopen{}\left(X\right)\mathclose{}}) = 1\).

Example 42 (Uniqueness in the best-predictor exercise) In Exercise 10, \(f_2\mathopen{}\left(X\right)\mathclose{} = 4X\) differs from \({f^*\mathopen{}\left(X\right)\mathclose{}}\) when \(X= 0\), which has probability \(0.5\). So \(f_2\) is not a minimizer, which matches its risk \(2\) being larger than \(1\).

4 Entropy and cross-entropy

Definition 22 (Surprisal (self-information, information content)) The surprisal of a possible value \(x\) of a discrete random variable \(X\) with \(\mathbb{P}(X=x) > 0\) is

\[{\mathbb{s}\mathopen{}\left(x\right)\mathclose{}} \stackrel{\text{def}}{=}-\log \mathbb{P}(X=x).\]

The surprisal is also called the self-information [16, p. 73] or the information content [17, p. 49] of \(x\). With the natural logarithm, the surprisal is measured in nats, and with base 2 in bits. Evaluating \({\mathbb{s}\mathopen{}\left(\cdot\right)\mathclose{}}\) at the random variable \(X\) itself gives the random variable \({\mathbb{s}\mathopen{}\left(X\right)\mathclose{}}\), which equals \({\mathbb{s}\mathopen{}\left(x\right)\mathclose{}}\) when \(X = x\). The notation leaves the probability mass function of \(X\) implicit.

Example 43 (Surprisal of the values of a biased coin flip) Let \(X\) be \(1\) if one flip of a coin lands heads and \(0\) if it lands tails, with \(\mathbb{P}(X=1) = \tfrac{1}{4}\) and \(\mathbb{P}(X=0) = \tfrac{3}{4}\). Using logarithms to base 2, so that surprisal is measured in bits, the surprisal of the value \(1\) (heads) is \({\mathbb{s}\mathopen{}\left(1\right)\mathclose{}} = -\log_2 \tfrac{1}{4} = 2\) bits, and the surprisal of the value \(0\) (tails) is \({\mathbb{s}\mathopen{}\left(0\right)\mathclose{}} = -\log_2 \tfrac{3}{4} = \log_2 \tfrac{4}{3} \approx 0.415\) bits. For a fair coin, each value of \(X\) has surprisal \(-\log_2 \tfrac{1}{2} = 1\) bit.

Remark 10 (Rarer values are more surprising). A possible value \(x\) with probability \(\mathbb{P}(X=x) = 1\) has surprisal \(0\), and the surprisal grows without bound as \(\mathbb{P}(X=x) \to 0\).

Definition 23 (Surprisal of a value of a continuous random variable) The surprisal of a possible value \(x\) of a continuous random variable \(X\) with probability density function \(\mathbb{p}(X=x)\) and \(\mathbb{p}(X=x) > 0\) is

\[{\mathbb{s}\mathopen{}\left(x\right)\mathclose{}} \stackrel{\text{def}}{=}-\log \mathbb{p}(X=x).\]

This extends the surprisal of a value of a discrete random variable, with the density in place of the probability.

Example 44 (Surprisal of a value of a uniform random variable) Let \(X\) be uniformly distributed on the interval from \(0\) to \(a > 0\), so that its density is \(\mathbb{p}(X=x) = 1/a\) for \(0 < x < a\). Every value in the interval has surprisal

\[{\mathbb{s}\mathopen{}\left(x\right)\mathclose{}} = -\log \tfrac{1}{a} = \log a.\]

With \(a = \tfrac{1}{2}\) and logarithms to base 2, the surprisal of every value is \(\log_2 \tfrac{1}{2} = -1\) bit. A density value can exceed \(1\), so unlike the surprisal of a discrete value, it can be negative.

Definition 24 (Entropy (differential entropy, for continuous random variables)) The entropy of a random variable \(X\) is the expectation of its surprisal (discrete or continuous) [18], [19, Sec. 8.1], when that expectation exists:

\[{\mathbb{H}\mathopen{}\left(X\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[{\mathbb{s}\mathopen{}\left(X\right)\mathclose{}}\right]\mathclose{}}.\]

For a continuous random variable, the entropy is also called the differential entropy.

Theorem 15 (Entropy of a discrete random variable) Let \(X\) be a discrete random variable with probability mass function \(\mathbb{P}(X=x)\). Then

\[{\mathbb{H}\mathopen{}\left(X\right)\mathclose{}} = -\sum_{x \in \mathcal{R}(X)} \mathbb{P}(X=x) \log \mathbb{P}(X=x).\]

A term with \(\mathbb{P}(X=x) = 0\) is taken to be \(0\), because \(p \log p \to 0\) as \(p \to 0\).

Proof. Let \(g(x) \stackrel{\text{def}}{=}{\mathbb{s}\mathopen{}\left(x\right)\mathclose{}} = -\log \mathbb{P}(X=x)\), which is finite for each value \(x\) with \(\mathbb{P}(X=x) > 0\). Then LOTUS gives

\[ \begin{aligned} {\mathbb{H}\mathopen{}\left(X\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[{\mathbb{s}\mathopen{}\left(X\right)\mathclose{}}\right]\mathclose{}} && \text{(definition of entropy)} \\ &= {\mathbb{E}\mathopen{}\left[g(X)\right]\mathclose{}} && \text{(definition of } g \text{)} \\ &= \sum_{x \in \mathcal{R}(X)} g(x) \cdot\mathbb{P}(X=x) && \text{(LOTUS)} \\ &= -\sum_{x \in \mathcal{R}(X)} \mathbb{P}(X=x) \log \mathbb{P}(X=x). && \text{(definition of } g \text{)} \end{aligned} \]

Theorem 16 (Entropy of a continuous random variable) Let \(X\) be a continuous random variable with probability density function \(\mathbb{p}(X=x)\). Then

\[{\mathbb{H}\mathopen{}\left(X\right)\mathclose{}} = -\int_{x \in \mathcal{R}(X)} \mathbb{p}(X=x) \log \mathbb{p}(X=x) \, dx.\]

The integrand is taken to be \(0\) where \(\mathbb{p}(X=x) = 0\), because \(p \log p \to 0\) as \(p \to 0\).

Proof. Let \(g(x) \stackrel{\text{def}}{=}{\mathbb{s}\mathopen{}\left(x\right)\mathclose{}} = -\log \mathbb{p}(X=x)\). Then LOTUS gives

\[ \begin{aligned} {\mathbb{H}\mathopen{}\left(X\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[{\mathbb{s}\mathopen{}\left(X\right)\mathclose{}}\right]\mathclose{}} && \text{(definition of entropy)} \\ &= {\mathbb{E}\mathopen{}\left[g(X)\right]\mathclose{}} && \text{(definition of } g \text{)} \\ &= \int_{x \in \mathcal{R}(X)} g(x) \cdot\mathbb{p}(X=x) \, dx && \text{(LOTUS)} \\ &= -\int_{x \in \mathcal{R}(X)} \mathbb{p}(X=x) \log \mathbb{p}(X=x) \, dx. && \text{(definition of } g \text{)} \end{aligned} \]

Example 45 (Entropy of a uniform random variable) Let \(X\) be uniformly distributed on the interval from \(0\) to \(a > 0\), as in Example 44. By Theorem 16,

\[ \begin{aligned} {\mathbb{H}\mathopen{}\left(X\right)\mathclose{}} &= -\int_0^a \tfrac{1}{a} \log \tfrac{1}{a} \, dx \\ &= \log a. \end{aligned} \]

Every value has the same surprisal \(\log a\), so its expectation is also \(\log a\). With \(a = \tfrac{1}{2}\) and logarithms to base 2, the entropy is \(-1\) bit. The entropy of a continuous random variable can be negative [19], Example 8.1.1.

Remark 11 (The base of the logarithm is a unit). With the natural logarithm, entropy is measured in nats. With base 2, it is measured in bits. Changing the base multiplies every entropy by the same positive constant. It does not change which distribution has the larger entropy.

Example 46 (Entropy of a coin flip) Let \(X\) be the result of one flip of a coin with \(\mathbb{P}(X=1) = \pi\) and \(\mathbb{P}(X=0) = 1 - \pi\) (Bernoulli distribution). By Theorem 15, using logarithms to base 2,

\[H(X) = -\pi\log_2 \pi- (1 - \pi) \log_2 (1 - \pi).\]

For a fair coin, \(\pi= 1/2\), and

\[ \begin{aligned} H(X) &= -\tfrac{1}{2} \log_2 \tfrac{1}{2} - \tfrac{1}{2} \log_2 \tfrac{1}{2} \\ &= \tfrac{1}{2} + \tfrac{1}{2} \\ &= 1 \text{ bit}. \end{aligned} \]

For a coin that always lands heads, \(\pi= 1\), and

\[ \begin{aligned} H(X) &= -1 \cdot \log_2 1 - 0 \\ &= 0. \end{aligned} \]

The outcome is known in advance, so there is no uncertainty to measure. An entropy of \(0\) means that all of the probability is on one outcome.

Remark 12 (Where the discrete formula comes from). Murphy [20, Sec. 2.8.1], p. 56 gives the discrete formula of Theorem 15 as the definition, and notes that among distributions on \(K\) outcomes it is largest, \(\log K\), for the uniform distribution.

Definition 25 (Cross-entropy) Let \(X\) be a random variable with probability mass function or density \(p\), and let \(q\) be a mass function or density on the same values. The cross-entropy of \(q\) relative to \(p\) [21] is the expectation, when \(X\) has distribution \(p\), of the surprisal (or its continuous form) \(\mathbb{s}_q\) measured by \(q\), where \({\mathbb{s}_{q}\mathopen{}\left(x\right)\mathclose{}} \stackrel{\text{def}}{=}-\log q\mathopen{}\left(x\right)\mathclose{}\):

\[{\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[{\mathbb{s}_{q}\mathopen{}\left(X\right)\mathclose{}}\right]\mathclose{}}.\]

We take \(-\log 0 = \infty\), so when \(\mathbb{P}(q(X) = 0) > 0\), the cross-entropy is infinite.

Exercise 11 (Mass function of the negative log probability for two mass functions) Let \(X\) have probability mass function \(p = (\tfrac{1}{2}, \tfrac{1}{4}, \tfrac{1}{4})\) on \({\mathopen{}\left\{1, 2, 3\right\}\mathclose{}}\), and let \(q = (\tfrac{1}{4}, \tfrac{1}{2}, \tfrac{1}{4})\) be another probability mass function on \({\mathopen{}\left\{1, 2, 3\right\}\mathclose{}}\). Write \(q_X\) for the random variable that equals \(q_k\) when \(X = k\).

Find the probability mass function of \(Y = -\log_2 q_X\).

Solution 7. The variable \(Y\) equals \(-\log_2 \tfrac{1}{4} = 2\) when \(X \in {\mathopen{}\left\{1, 3\right\}\mathclose{}}\), and \(-\log_2 \tfrac{1}{2} = 1\) when \(X = 2\). So \(\mathbb{P}(Y = 2) = p_1 + p_3 = \tfrac{3}{4}\) and \(\mathbb{P}(Y = 1) = p_2 = \tfrac{1}{4}\).

Exercise 12 (Expected negative log probability for two mass functions) Using the mass function of \(Y = -\log_2 q_X\) from Exercise 11, compute \({\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}}\).

Solution 8. From the mass function in Solution 7,

\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[Y\right]\mathclose{}} &= 2 \cdot\tfrac{3}{4} + 1 \cdot\tfrac{1}{4} \\ &= \tfrac{7}{4}. \end{aligned} \]

Exercise 13 (Cross-entropy of two mass functions from the definition) Let \(p = (\tfrac{1}{2}, \tfrac{1}{4}, \tfrac{1}{4})\) and \(q = (\tfrac{1}{4}, \tfrac{1}{2}, \tfrac{1}{4})\) be probability mass functions on \({\mathopen{}\left\{1, 2, 3\right\}\mathclose{}}\).

Compute the cross-entropy \({\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}}\) in bits from its definition.

Solution 9. From the definition,

\[ \begin{aligned} {\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} &= -\tfrac{1}{2} \log_2 \tfrac{1}{4} - \tfrac{1}{4} \log_2 \tfrac{1}{2} - \tfrac{1}{4} \log_2 \tfrac{1}{4} \\ &= \tfrac{1}{2} \cdot 2 + \tfrac{1}{4} \cdot 1 + \tfrac{1}{4} \cdot 2 \\ &= \tfrac{7}{4}. \end{aligned} \]

Exercise 14 (Cross-entropy and the expected negative log probability) Compare your value of \({\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}}\) from Exercise 13 with your value of \({\mathbb{E}\mathopen{}\left[-\log_2 q_X\right]\mathclose{}}\) from Exercise 12, where \(X\) has probability mass function \(p\).

Solution 10. The cross-entropy equals \({\mathbb{E}\mathopen{}\left[-\log_2 q_X\right]\mathclose{}}\). The two calculations group the same three terms differently: the expectation first combines outcomes 1 and 3, which have the same value of \(-\log_2 q_X\).

Exercise 15 (Cross-entropy as a sum when \(q\) is positive where \(p\) is) Let \(p\) and \(q\) be probability mass functions on \({\mathopen{}\left\{1, \ldots, K\right\}\mathclose{}}\) with \(q_k > 0\) whenever \(p_k > 0\). Let \(X\) have mass function \(p\), and write \(q_X\) for the random variable that equals \(q_k\) when \(X = k\). Show that the cross-entropy, the expectation \({\mathbb{E}\mathopen{}\left[-\log q_X\right]\mathclose{}}\), equals \(-\sum_{k=1}^{K} p_k \log q_k\).

Solution 11. Let \(g(k) \stackrel{\text{def}}{=}-\log q_k\), so that \(-\log q_X = g(X)\). Since \(q_k > 0\) whenever \(p_k > 0\), \(g(k)\) is finite for every value \(k\) that \(X\) takes with positive probability. By LOTUS,

\[ \begin{aligned} {\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[-\log q_X\right]\mathclose{}} \\ &= {\mathbb{E}\mathopen{}\left[g(X)\right]\mathclose{}} \\ &= \sum_{k : p_k > 0} g(k) \cdot\mathbb{P}(X = k) \\ &= -\sum_{k : p_k > 0} p_k \log q_k \\ &= -\sum_{k=1}^{K} p_k \log q_k. \end{aligned} \]

The last line adds the terms with \(p_k = 0\), which are \(0\) by the convention in Theorem 17.

Exercise 16 (Cross-entropy when \(q\) gives zero probability to a possible outcome) Let \(p\) and \(q\) be probability mass functions on \({\mathopen{}\left\{1, \ldots, K\right\}\mathclose{}}\) with \(q_k = 0\) for some \(k\) with \(p_k > 0\). Let \(X\) have mass function \(p\), and write \(q_X\) for the random variable that equals \(q_k\) when \(X = k\).

Find the cross-entropy \({\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}}\).

Solution 12. By Definition 25, \({\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} = \infty\).

Exercise 17 (Expected negative log probability when \(q\) gives zero probability to a possible outcome) In the setting of Exercise 16, find the expectation \({\mathbb{E}\mathopen{}\left[-\log q_X\right]\mathclose{}}\).

Solution 13. Here \(-\log q_X \ge 0\) always, and \(-\log q_X = \infty\) with probability at least \(p_k > 0\), so \({\mathbb{E}\mathopen{}\left[-\log q_X\right]\mathclose{}} = \infty\).

Theorem 17 (Cross-entropy of two mass functions) Let \(p\) and \(q\) be probability mass functions on \({\mathopen{}\left\{1, \ldots, K\right\}\mathclose{}}\), with \(p_k\) and \(q_k\) the probabilities of outcome \(k\). Then the cross-entropy is

\[{\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} = -\sum_{k=1}^{K} p_k \log q_k.\]

A term with \(p_k = 0\) is taken to be \(0\) whatever \(q_k\) is, as for entropy.

Proof. Let \(X\) have mass function \(p\), and let \(g(k) \stackrel{\text{def}}{=}-\log q_k\), so that \(-\log q(X) = g(X)\). First suppose \(q_k > 0\) whenever \(p_k > 0\). Then \(g(k)\) is finite for every value \(k\) that \(X\) takes with positive probability, and LOTUS gives

\[ \begin{aligned} {\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[g(X)\right]\mathclose{}} && \text{(definition)} \\ &= \sum_{k : p_k > 0} g(k) \cdot\mathbb{P}(X = k) && \text{(LOTUS)} \\ &= -\sum_{k : p_k > 0} p_k \log q_k && \text{(definition of } g \text{ and } \mathbb{P}(X = k) = p_k \text{)} \\ &= -\sum_{k=1}^{K} p_k \log q_k. && \text{(terms with } p_k = 0 \text{ are } 0 \text{)} \end{aligned} \]

Now suppose \(q_k = 0\) for some \(k\) with \(p_k > 0\). Then \(-\log q(X) \ge 0\), and \(-\log q(X) = \infty\) with probability at least \(p_k > 0\), so \({\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} = \infty\). The right side is also \(\infty\), because its term for \(k\) is \(-p_k \log 0 = \infty\) and its other terms are finite or \(\infty\) but never \(-\infty\).

Theorem 18 (Cross-entropy of two densities) Let \(X\) be a continuous random variable with density \(p\) on its range \(\mathcal{R}(X)\), and let \(q\) be a density on \(\mathcal{R}(X)\). Then the cross-entropy is

\[{\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} = -\int_{x \in \mathcal{R}(X)} p(x) \log q(x) \, dx.\]

The integrand is taken to be \(0\) where \(p(x) = 0\). If \(\mathbb{P}(q(X) = 0) > 0\), both sides are \(\infty\).

Proof. Let \(g(x) \stackrel{\text{def}}{=}-\log q(x)\), so that \(-\log q(X) = g(X)\). Then LOTUS gives

\[ \begin{aligned} {\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[g(X)\right]\mathclose{}} && \text{(definition)} \\ &= \int_{x \in \mathcal{R}(X)} g(x) \cdot p(x) \, dx && \text{(LOTUS)} \\ &= -\int_{x \in \mathcal{R}(X)} p(x) \log q(x) \, dx. && \text{(definition of } g \text{)} \end{aligned} \]

Now suppose \(\mathbb{P}(q(X) = 0) > 0\). Then \(-\log q(X) \ge 0\), and \(-\log q(X) = \infty\) with positive probability, so \({\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} = \infty\). The integrand \(-p(x) \log q(x)\) is \(\infty\) wherever \(q(x) = 0\) and \(p(x) > 0\), which is a region of positive probability under \(p\), and it is never \(-\infty\), so the right side is also \(\infty\).

Example 47 (Cross-entropy of two densities on the unit interval) Let \(p(x) = 1\) and \(q(x) = 2x\) for \(0 < x < 1\) be densities. By Theorem 18, using the natural logarithm,

\[ \begin{aligned} {\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} &= -\int_0^1 \log (2x) \, dx \\ &= -\log 2 - \int_0^1 \log x \, dx \\ &= -\log 2 + 1 \\ &\approx 0.307 \text{ nats}. \end{aligned} \]

Exercise 18 (Cross-entropy of a mass function with itself) Let \(p = (\tfrac{1}{2}, \tfrac{1}{4}, \tfrac{1}{4})\) be a probability mass function on \({\mathopen{}\left\{1, 2, 3\right\}\mathclose{}}\). Compute the cross-entropy \({\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}}\) in bits.

Solution 14. \[ \begin{aligned} {\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}} &= -\tfrac{1}{2} \log_2 \tfrac{1}{2} - \tfrac{1}{4} \log_2 \tfrac{1}{4} - \tfrac{1}{4} \log_2 \tfrac{1}{4} \\ &= \tfrac{1}{2} \cdot 1 + \tfrac{1}{4} \cdot 2 + \tfrac{1}{4} \cdot 2 \\ &= \tfrac{3}{2} \text{ bits}. \end{aligned} \]

Exercise 19 (Entropy of a random variable with a given mass function) Let \(X\) be a discrete random variable with probability mass function \(p = (\tfrac{1}{2}, \tfrac{1}{4}, \tfrac{1}{4})\) on \({\mathopen{}\left\{1, 2, 3\right\}\mathclose{}}\).

Compute the entropy \({\mathbb{H}\mathopen{}\left(X\right)\mathclose{}}\) in bits from its definition.

Solution 15. The range of \(X\) is \({\mathopen{}\left\{1, 2, 3\right\}\mathclose{}}\), and \(\mathbb{P}(X = k) = p_k\), so

\[ \begin{aligned} {\mathbb{H}\mathopen{}\left(X\right)\mathclose{}} &= -\sum_{k \in \mathcal{R}(X)} \mathbb{P}(X = k) \log_2 \mathbb{P}(X = k) \\ &= -\tfrac{1}{2} \log_2 \tfrac{1}{2} - \tfrac{1}{4} \log_2 \tfrac{1}{4} - \tfrac{1}{4} \log_2 \tfrac{1}{4} \\ &= \tfrac{3}{2} \text{ bits}. \end{aligned} \]

Exercise 20 (Entropy and the cross-entropy of a mass function with itself) Compare your value of \({\mathbb{H}\mathopen{}\left(X\right)\mathclose{}}\) from Exercise 19 with your value of \({\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}}\) from Exercise 18.

Solution 16. The entropy equals \({\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}}\), because the two sums have the same terms.

Exercise 21 (Entropy as a cross-entropy, for any mass function) Let \(X\) be a discrete random variable with probability mass function \(p\) on \({\mathopen{}\left\{1, \ldots, K\right\}\mathclose{}}\). Show that \({\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}} = {\mathbb{H}\mathopen{}\left(X\right)\mathclose{}}\).

Solution 17. By definition, \({\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[-\log p(X)\right]\mathclose{}}\). Since \(\mathbb{P}(X = k) = p_k\), we have \(-\log p(X) = {\mathbb{s}\mathopen{}\left(X\right)\mathclose{}}\), the surprisal of \(X\). So \({\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[{\mathbb{s}\mathopen{}\left(X\right)\mathclose{}}\right]\mathclose{}} = {\mathbb{H}\mathopen{}\left(X\right)\mathclose{}}\) by the definition of entropy.

Corollary 6 (Entropy is the cross-entropy of a distribution with itself) Let \(X\) be a random variable with probability mass function or density \(p\). Then

\[{\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}} = {\mathbb{H}\mathopen{}\left(X\right)\mathclose{}}.\]

Proof. By the definition of cross-entropy with \(q = p\), \({\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[-\log p(X)\right]\mathclose{}}\). The random variable \(-\log p(X)\) is the surprisal (or its continuous form) \({\mathbb{s}\mathopen{}\left(X\right)\mathclose{}}\). So \({\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[{\mathbb{s}\mathopen{}\left(X\right)\mathclose{}}\right]\mathclose{}} = {\mathbb{H}\mathopen{}\left(X\right)\mathclose{}}\) by the definition of entropy.

Remark 13 (Cross-entropy is an expected surprisal). By definition, the cross-entropy \({\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}}\) is the average surprisal of the values drawn from \(p\), with each surprisal measured by \(q\) as \(-\log q(x)\). By Corollary 6, measuring with \(p\) itself gives the entropy.

Definition 26 (Kullback–Leibler divergence (KL divergence, relative entropy)) For \(p\) and \(q\) as in Definition 25, the Kullback–Leibler divergence (KL divergence, or relative entropy) of \(q\) from \(p\) is

\[{{D_{\mathrm{KL}}}\mathopen{}\left(p \,\Vert\, q\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} - {\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}}.\]

Theorem 19 (KL divergence of two mass functions) Let \(p\) and \(q\) be probability mass functions on \({\mathopen{}\left\{1, \ldots, K\right\}\mathclose{}}\), with \(p_k\) and \(q_k\) the probabilities of outcome \(k\), and suppose \(q_k > 0\) whenever \(p_k > 0\). Then the KL divergence is

\[{{D_{\mathrm{KL}}}\mathopen{}\left(p \,\Vert\, q\right)\mathclose{}} = \sum_{k=1}^{K} p_k \log \frac{p_k}{q_k}.\]

A term with \(p_k = 0\) is taken to be \(0\).

Proof. By the discrete formula for cross-entropy,

\[ \begin{aligned} {{D_{\mathrm{KL}}}\mathopen{}\left(p \,\Vert\, q\right)\mathclose{}} &= {\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} - {\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}} && \text{(definition)} \\ &= -\sum_{k=1}^{K} p_k \log q_k + \sum_{k=1}^{K} p_k \log p_k && \text{(discrete formula, twice)} \\ &= \sum_{k=1}^{K} p_k \mathopen{}\left(\log p_k - \log q_k\right)\mathclose{} && \text{(combine the sums)} \\ &= \sum_{k=1}^{K} p_k \log \frac{p_k}{q_k}. && \text{(difference of logarithms)} \end{aligned} \]

Remark 14 (The divergence is not symmetric). In general, \({{D_{\mathrm{KL}}}\mathopen{}\left(p \,\Vert\, q\right)\mathclose{}} \ne {{D_{\mathrm{KL}}}\mathopen{}\left(q \,\Vert\, p\right)\mathclose{}}\), so KL divergence is not a distance between \(p\) and \(q\).

Example 48 (Cross-entropy and divergence for two coins) Let \(p = (\tfrac{1}{2}, \tfrac{1}{2})\) be a fair coin and \(q = (\tfrac{1}{4}, \tfrac{3}{4})\) a coin biased toward the second outcome. Using logarithms to base 2, \(\log_2 \tfrac{1}{4} = -2\) and \(\log_2 \tfrac{3}{4} = \log_2 3 - 2\), so

\[ \begin{aligned} {\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} &= -\tfrac{1}{2} \log_2 \tfrac{1}{4} - \tfrac{1}{2} \log_2 \tfrac{3}{4} \\ &= 1 + \tfrac{1}{2} \mathopen{}\left(2 - \log_2 3\right)\mathclose{} \\ &= 2 - \tfrac{1}{2} \log_2 3 \\ &\approx 1.208, \\ {\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}} &= -\tfrac{1}{2} \log_2 \tfrac{1}{2} - \tfrac{1}{2} \log_2 \tfrac{1}{2} \\ &= 1, \\ {{D_{\mathrm{KL}}}\mathopen{}\left(p \,\Vert\, q\right)\mathclose{}} &= {\mathbb{H}\mathopen{}\left(p, q\right)\mathclose{}} - {\mathbb{H}\mathopen{}\left(p, p\right)\mathclose{}} = 1 - \tfrac{1}{2} \log_2 3 \approx 0.208. \end{aligned} \]

Here the cross-entropy is larger than the entropy of \(p\), and the divergence is the difference.

Remark 15 (Where the definitions come from). Murphy [20, Sec. 2.8.2], pp. 57–58 defines the KL divergence and the cross-entropy as above, and gives the coding interpretation: the cross-entropy is the average number of bits needed to encode data from a source with distribution \(p\) using a code designed for \(q\).

5 Further reading

Another explanation

Lange’s textbook [22] covers conditioning as a tool for computing expectations in Section 2.3 and tail probability methods in Section 2.5.

References

[1]

Wikipedia contributors, “Expected value — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Expected_value

[2]

P. Billingsley, Probability and measure, 3rd ed. in Wiley series in probability and mathematical statistics. New York: Wiley, 1995.

[3]

J. Soch, “Proof: Expected value of a non-negative random variable.” The Book of Statistical Proofs, 2020. doi: 10.5281/zenodo.4305949.

[4]

Wikipedia contributors, “Law of the unconscious statistician — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Law_of_the_unconscious_statistician

[5]

A. Gut, Probability: A graduate course, 2nd ed. in Springer texts in statistics. New York: Springer, 2013. doi: 10.1007/978-1-4614-4708-5.

[6]

G. Casella and R. Berger, Statistical inference, 2nd ed. Cengage Learning, 2002. Available: https://www.cengage.com/c/statistical-inference-2e-casella-berger/9780534243128/

[7]

Wikipedia contributors, “Law of the unconscious statistician — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Law_of_the_unconscious_statistician

[8]

A. J. Dobson and A. G. Barnett, An introduction to generalized linear models, 4th ed. CRC press, 2018. doi: 10.1201/9781315182780.

[9]

Wikipedia contributors, “Conditional expectation — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Conditional_expectation

[10]

K. Ross, “An introduction to probability and simulation.” Bookdown, 2022. Available: https://bookdown.org/kevin_davisross/probsim-book/

[11]

W. Rudin, Principles of mathematical analysis, 3rd ed. in International series in pure and applied mathematics. New York: McGraw-Hill, 1976.

[12]

Wikipedia contributors, “Loss function — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Loss_function

[13]

T. Hastie, R. Tibshirani, and J. Friedman, The elements of statistical learning: Data mining, inference, and prediction, 2nd ed. Springer, 2009. doi: 10.1007/978-0-387-84858-7.

[14]

Wikipedia contributors, “Risk function — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Risk_function

[15]

G. James, D. Witten, T. Hastie, and R. Tibshirani, An introduction to statistical learning: With applications in R, 2nd ed. Springer, 2021. doi: 10.1007/978-1-0716-1418-1.

[16]

I. Goodfellow, Y. Bengio, and A. Courville, Deep learning. Cambridge, MA: MIT Press, 2016. Available: https://www.deeplearningbook.org/

[17]

C. M. Bishop, Pattern recognition and machine learning. in Information science and statistics. New York: Springer, 2006.

[18]

Wikipedia contributors, “Entropy (information theory) — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Entropy_(information_theory)

[19]

T. M. Cover and J. A. Thomas, Elements of information theory, 2nd ed. Wiley-Interscience, 2006. doi: 10.1002/047174882X.

[20]

K. P. Murphy, Machine learning: A probabilistic perspective. in Adaptive computation and machine learning. MIT Press, 2012.

[21]

Wikipedia contributors, “Cross-entropy — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Cross-entropy

[22]

K. Lange, Applied probability, 3rd ed. in Springer texts in statistics. Springer, 2024. doi: 10.1007/978-1-0716-4172-9.