Random variables, distributions, expectation, and prediction
Last modified: 2026-10-25
Source and scope
This appendix fixes the probabilistic tools used across classification, loss functions, and noise models. It points to the worked calculations, interactive widgets, and exercises in the lab’s shared prerequisite site: Probability for Data Science.
The joint probability table, chain rule, continuous density intuition, and conditional independence examples are adapted from Brian Hutchinson’s Fall 2024 DATA 471 probability refresher [1].
Everything here is standard. Goodfellow et al. [2], chapter 3, covers probability and information theory at length and is freely readable online: random variables, probability distributions, marginal and conditional distributions, expectation, variance, covariance, and Bayes’ rule. Bishop [3] is a second account of the same material, with probability in section 1.2.
Probability provides the mathematical framework for quantifying uncertainty, modeling noise in data, and deriving optimal predictions under loss functions. This appendix summarizes the key concepts and links directly to their full definitions and worked exercises in Probability for Data Science.
Probability for Data Science covers the probabilistic foundations these notes assume:
Worked exercises in Probability for Data Science:
Watch
Read
GeeksforGeeks explains:
Watch
5MinuteAI’s Probability & Statistics playlist [11] follows the University of Washington’s CSE 312. Its five-minute videos cover:
Read
The University of Washington’s CSE 446 slides open with a probability review [12], Lecture 1.
Theorem 1 (The chain rule of probability)
Theorem 1 (The chain rule of probability) For random variables \(X_1, \dots, X_N\), in any order, \[ \Pr(X_1, \dots, X_N) = \prod_{i=1}^N \Pr(X_i \mid X_1, \dots, X_{i-1}). \] Each factor conditions on all the variables that precede it in the chosen order [13].
Proof
Proof. For \(N = 2\), this is the law of conditional probability, \(\Pr(X_1, X_2) = \Pr(X_1)\Pr(X_2 \mid X_1)\). Suppose it holds for \(N - 1\) variables. Apply the \(N = 2\) case to the pair \((X_1, \dots, X_{N-1})\) and \(X_N\): \[ \Pr(X_1, \dots, X_N) = \Pr(X_1, \dots, X_{N-1})\,\Pr(X_N \mid X_1, \dots, X_{N-1}). \] Then expand the first factor by the induction hypothesis.
Example 1 (Two factorizations of the same spam probabilities)
Example 1 (Two factorizations of the same spam probabilities) In the Spambase data [14], let \(Y\) be the class (spam or not), \(X_1\) be “the email contains the word free”, and \(X_2\) be “the email contains the word money”. Take the proportions of the 4601 emails as probabilities. Compute \(\Pr(X_1 = 1, X_2 = 1, Y = 1)\) directly, and as the product of Theorem 1 in two different orders:
import pandas as pd
sb_cr = pd.read_csv("data/spambase.csv")
y_cr = sb_cr["Class"] == 1
x1_cr = sb_cr["word_freq_free"] > 0
x2_cr = sb_cr["word_freq_money"] > 0
def pr_cr(event):
"""The proportion of emails for which the event holds."""
return event.mean()
direct_cr = pr_cr(x1_cr & x2_cr & y_cr)
# Order (Y, X1, X2): Pr(y) Pr(x1 | y) Pr(x2 | x1, y).
order_a_cr = pr_cr(y_cr) * (pr_cr(x1_cr & y_cr) / pr_cr(y_cr)) * (pr_cr(x1_cr & x2_cr & y_cr) / pr_cr(x1_cr & y_cr))
# Order (X2, Y, X1): Pr(x2) Pr(y | x2) Pr(x1 | y, x2).
order_b_cr = pr_cr(x2_cr) * (pr_cr(y_cr & x2_cr) / pr_cr(x2_cr)) * (pr_cr(x1_cr & y_cr & x2_cr) / pr_cr(y_cr & x2_cr))
print(direct_cr, order_a_cr, order_b_cr)0.09497935231471419 0.09497935231471419 0.09497935231471419sb_cr <- read.csv("data/spambase.csv")
y_cr <- sb_cr$Class == 1
x1_cr <- sb_cr$word_freq_free > 0
x2_cr <- sb_cr$word_freq_money > 0
# The proportion of emails for which the event holds.
pr_cr <- function(event) mean(event)
direct_cr <- pr_cr(x1_cr & x2_cr & y_cr)
# Order (Y, X1, X2): Pr(y) Pr(x1 | y) Pr(x2 | x1, y).
order_a_cr <- pr_cr(y_cr) * (pr_cr(x1_cr & y_cr) / pr_cr(y_cr)) * (pr_cr(x1_cr & x2_cr & y_cr) / pr_cr(x1_cr & y_cr))
# Order (X2, Y, X1): Pr(x2) Pr(y | x2) Pr(x1 | y, x2).
order_b_cr <- pr_cr(x2_cr) * (pr_cr(y_cr & x2_cr) / pr_cr(x2_cr)) * (pr_cr(x1_cr & y_cr & x2_cr) / pr_cr(y_cr & x2_cr))
c(direct = direct_cr, order_a = order_a_cr, order_b = order_b_cr) direct order_a order_b
0.09497935 0.09497935 0.09497935 All three equal 0.0950: about 9.5% of emails are spam that contains both words. The two orders give different factors, such as \(\Pr(x_1 \mid y)\) in one and \(\Pr(y \mid x_2)\) in the other, but the same product.
For three variables, this says \(\Pr(x, y, z) = \Pr(x)\Pr(y \mid x)\Pr(z \mid x, y)\). Reordering the variables gives a different factorization of the same joint distribution.
The chain rule is the backbone of autoregressive sequence models, which predict each element of a sequence from the elements before it, and of the directed graphical models that generalize them.
The chain rule is exact, and it has too many parameters to use as it stands: the last factor, \(\Pr(X_N \mid X_1, \dots, X_{N-1})\), is a separate distribution for every combination of the earlier variables. A modeling assumption cuts that down. Conditional independence says that, once some variable is known, other variables carry no further information about each other.
Definition 1 (The naive Bayes assumption)
Definition 1 (The naive Bayes assumption) Let \(Y\) be a class label and \(X_1, \dots, X_{\ninputvars}\) the input features. The naive Bayes assumption is that the features are conditionally independent given the class [15]: \[ \Pr(X_1, \dots, X_{\ninputvars} \mid Y) = \prod_{j=1}^{\ninputvars} \Pr(X_j \mid Y). \] A naive Bayes classifier combines this with Bayes’ rule, \(\Pr(Y \mid X_1, \dots, X_{\ninputvars}) \propto \Pr(Y)\prod_{j=1}^{\ninputvars} \Pr(X_j \mid Y)\) [16].
Example 2 (Do free and money appear independently in spam?)
Example 2 (Do free and money appear independently in spam?) Continue with the Spambase events of the chain rule example, restricted to the spam emails (\(Y = 1\)). Compare the joint probability \(\Pr(X_1 = 1, X_2 = 1 \mid Y = 1)\) with the product \(\Pr(X_1 = 1 \mid Y = 1)\Pr(X_2 = 1 \mid Y = 1)\) that the naive Bayes assumption would give:
0.2410369553226696 0.20490275387446608Among spam, 54.6% of emails contain free and 37.6% contain money, so independence would predict 20.5% with both. The data show 24.1%. The assumption is wrong here, since the two words tend to appear together, which is why it is called naive. A naive Bayes classifier uses the product anyway, and often classifies well despite the error.
The assumption is rarely exactly true, which is why it is called naive. Its value is in how few parameters it needs.
Example 3 (Counting parameters)
Example 3 (Counting parameters) Suppose \(Y\) takes \(K\) values and each of \(\ninputvars\) features is binary.
Without the naive Bayes assumption, \(\Pr(X_1, \dots, X_{\ninputvars} \mid Y = k)\) is an arbitrary distribution over \(2^{\ninputvars}\) outcomes, for each class \(k\). It needs \(2^{\ninputvars} - 1\) free numbers per class, since the probabilities must sum to one. With the assumption, each \(\Pr(X_j \mid Y = k)\) is a single Bernoulli probability, so each class needs only \(\ninputvars\) numbers.
# The number of free parameters in the class-conditional distributions
# of K classes with D binary features, with and without the assumption.
k_classes <- 2
d_features <- c(5, 20, 30)
full <- k_classes * (2^d_features - 1)
naive <- k_classes * d_features
data.frame(
D = d_features,
full = format(full, scientific = FALSE),
naive = naive
)With 20 binary features and 2 classes, the full model has about 2,097,150 parameters, more than most datasets have observations. The naive Bayes model has 40. The number grows exponentially in \(\ninputvars\) without the assumption, and linearly in \(\ninputvars\) with it.
Probability for Data Science covers moments, variance properties, and noise models:
Worked exercises in Probability for Data Science:
Watch
Read
GeeksforGeeks explains:
Watch
5MinuteAI explains expectation, variance and covariance and correlation [11].
Exercise 1 (Predicting with one number)
Exercise 1 (Predicting with one number) A city must predict the crime rate \(Y\) of a randomly chosen neighbourhood, using no input at all: it predicts the same number \(c\) for every neighbourhood. Write \(\mean \eqdef \Expf{Y}\).
Write the expected squared error of the prediction \(c\), as a function of \(c\).
Solution 1. The squared error of the prediction is \((\Outvar - c)^2\), so its expectation is \(\Expf{(\Outvar - c)^2}\).
Exercise 2 (Predicting with one number (part 2))
Exercise 2 (Predicting with one number (part 2)) Use the setup of Exercise 1.
Show that this expected squared error equals \(\Var{\Outvar} + (\mean - c)^2\).
Hint
Inside the square, add and subtract \(\mean\).
Solution 2. Add and subtract \(\mean\) inside the square:
\[ \Expf{(\Outvar - c)^2} = \Expf{\paren{(\Outvar - \mean) + (\mean - c)}^2} . \]
Expand the square, using the square of a sum:
\[ = \Expf{(\Outvar - \mean)^2 + 2 (\Outvar - \mean)(\mean - c) + (\mean - c)^2} . \]
Apply linearity of expectation. The numbers \(\mean\) and \(c\) are constants, so \(2(\mean - c)\) and \((\mean - c)^2\) come outside the expectation:
\[ = \Expf{(\Outvar - \mean)^2} + 2 (\mean - c)\, \Expf{\Outvar - \mean} + (\mean - c)^2 . \]
The middle expectation is zero, because
\[ \begin{aligned} \Expf{\Outvar - \mean} &= \Expf{\Outvar} - \mean \\ &= \mean - \mean \\ &= 0, \end{aligned} \]
so the risk reduces to:
\[ = \Expf{(\Outvar - \mean)^2} + (\mean - c)^2 . \]
The first term is the variance of \(\Outvar\):
\[ = \Var{\Outvar} + (\mean - c)^2 . \]
The probability notes state this result as a theorem.
Example 4 (Predicting fuel efficiency with a constant)
Example 4 (Predicting fuel efficiency with a constant) In the Auto data [20], treat the mpg values of the cars as the whole distribution of \(Y\), each car equally likely. For the constants \(c = 15, 20, 25\), compute the expected squared error \(\Expf{(Y - c)^2}\) directly and as \(\Var{Y} + (\Expf{Y} - c)^2\):
import numpy as np
import pandas as pd
y_cp = pd.read_csv("data/auto.csv")["mpg"].to_numpy(float)
mean_cp, var_cp = y_cp.mean(), y_cp.var() # the variance divides by n, as the distribution requires
for c in (15, 20, 25):
direct = np.mean((y_cp - c) ** 2)
decomposed = var_cp + (mean_cp - c) ** 2
print(c, round(direct, 3), round(decomposed, 3))15 132.096 132.096
20 72.637 72.637
25 63.178 63.178The two columns agree for every \(c\). The mean of mpg is 23.45 and its variance is 60.76, so the constant \(c = 20\) has expected squared error equal to the variance plus 11.87 for being 3.45 away from the mean. No constant can have a smaller expected squared error than the variance, as the next exercise shows.
Exercise 3 (The best single number)
Exercise 3 (The best single number) Use the decomposition of the expected squared error of a constant prediction.
Which number \(c\) minimizes \(\Expf{(\Outvar - c)^2}\)?
Solution 3. In the decomposition of the expected squared error of a constant prediction, the first term, \(\Var{\Outvar}\), does not depend on \(c\). The second term, \((\mean - c)^2\), is a square, so it is never negative, and it is zero exactly when \(c = \mean\). So
\[ \begin{aligned} c &= \mean \\ &= \Expf{\Outvar} \end{aligned} \]
minimizes the expected squared error, and no other number does. In the notation of the argmin, the set of minimizers has the single element \(\mean\).
Exercise 4 (The best single number (part 2))
Exercise 4 (The best single number (part 2)) Use the setup of Exercise 3.
What is the smallest expected squared error that any constant prediction can achieve?
Solution 4. At \(c = \mean\), the second term is zero, so the smallest expected squared error is \(\Var{\Outvar}\).
Under squared error, the best prediction made without any input is the mean, and the error that remains is the variance. A model that uses an input \(\predvar\) can only improve on \(\Var{\Outvar}\) by learning how the mean of \(\Outvar\) changes with \(\predvar\).
The probability notes state the result as a corollary.
Example 5 (Searching for the best constant)
Example 5 (Searching for the best constant) For the mpg values of the Auto data [20], find the constant \(c\) that minimizes the average squared error by numerical search, without using the mean:
The search ends at 23.446, the mean of mpg, and the smallest error it finds is 60.763, the variance. This matches the best constant prediction: the best constant under squared error is the mean, and the best error is the variance.
# HTML formats get an interactive plotly figure (hover for the values); the
# handout and other formats get the same curve with base R graphics.
# y_cp, mean_cp and var_cp come from the chunks of the example above.
# The expected squared error of the constant c, with each car equally likely.
squared_error <- function(c) sapply(c, function(ci) mean((y_cp - ci)^2))
grid_c <- seq(min(y_cp), max(y_cp), length.out = 300)
examples_c <- c(15, 20, 25)
if (knitr::is_html_output()) {
library(plotly)
widget <- plot_ly() |>
add_trace(
x = grid_c, y = squared_error(grid_c), type = "scatter", mode = "lines",
name = "expected squared error", line = list(color = "#1f77b4")
) |>
add_trace(
x = examples_c, y = squared_error(examples_c), type = "scatter",
mode = "markers", name = "the constants of the example",
marker = list(color = "#d95f02", size = 10)
) |>
add_trace(
x = mean_cp, y = var_cp, type = "scatter", mode = "markers",
name = "the mean, with the variance as its error",
marker = list(color = "#222222", size = 10, symbol = "diamond")
) |>
layout(
xaxis = list(title = "constant prediction c (mpg)"),
yaxis = list(title = "expected squared error", rangemode = "tozero"),
shapes = list(
list(
type = "line", x0 = mean_cp, x1 = mean_cp, y0 = 0, y1 = var_cp,
line = list(color = "#999999", dash = "dot")
),
list(
type = "line", x0 = min(grid_c), x1 = mean_cp, y0 = var_cp,
y1 = var_cp, line = list(color = "#999999", dash = "dot")
)
),
legend = list(orientation = "h", y = -0.25),
margin = list(t = 10)
) |>
config(displaylogo = FALSE, responsive = TRUE)
# The alt text: a plotly widget has none of its own.
htmltools::div(
role = "img",
`aria-label` = paste(
"Interactive plot of the expected squared error of a constant prediction",
"of mpg against the constant. The curve is a parabola with its lowest",
"point at the mean, where the error equals the variance."
),
widget
)
} else {
# The bottom margin leaves room under the plot for the legend.
op <- par(no.readonly = TRUE)
par(mar = c(7, 4, 1, 1))
plot(
grid_c, squared_error(grid_c), type = "l", col = "#1f77b4",
xlab = "constant prediction c (mpg)", ylab = "expected squared error"
)
points(examples_c, squared_error(examples_c), pch = 16, col = "#d95f02")
points(mean_cp, var_cp, pch = 18, cex = 1.5)
segments(mean_cp, 0, mean_cp, var_cp, lty = 3, col = "#999999")
segments(min(grid_c), var_cp, mean_cp, var_cp, lty = 3, col = "#999999")
# An empty plot over the whole device places the legend in the bottom margin.
par(fig = c(0, 1, 0, 1), mar = c(0, 0, 0, 0), new = TRUE)
plot.new()
legend(
"bottom", c("expected squared error", "constants of the example", "the mean"),
col = c("#1f77b4", "#d95f02", "black"), lty = c(1, NA, NA),
pch = c(NA, 16, 18), bty = "n", horiz = TRUE, cex = 0.8
)
par(op)
}mpg, for the Auto data of Example 5. The curve is a parabola, as the decomposition of the expected squared error of a constant prediction says: its lowest point is at the mean of mpg, and its height there is the variance. The dots are the three constants of Example 4.
Exercise 5 (The best constant for the five neighbourhoods)
Exercise 5 (The best constant for the five neighbourhoods) Take the five neighbourhoods of A First Model, and treat them as the whole population, each with probability \(1/5\). Then \(\Expf{(Y - c)^2}\) is the empirical risk of the model \(h(x) = c\), which ignores the income \(x\).
Use the best constant prediction to find the constant that minimizes this empirical risk, and the empirical risk it achieves. Compute both numbers.
Solution 5. By the best constant prediction, the best constant is the mean crime rate, and its empirical risk is the variance of the five crime rates, computed with divisor \(5\) rather than \(4\), because the five neighbourhoods are the whole population here.
0.288 0.023056 0.002721The best constant is \(c = 0.2880\), with empirical risk \(0.023056\).
Exercise 6 (The best constant for the five neighbourhoods (part 2))
Exercise 6 (The best constant for the five neighbourhoods (part 2)) Use the setup of Exercise 5.
Compare that empirical risk with the empirical risk of the fitted line of A First Model.
Solution 6. The fitted line reaches an empirical risk of \(0.002721\), which is 88% lower.
Exercise 7 (The best constant for the five neighbourhoods (part 3))
Exercise 7 (The best constant for the five neighbourhoods (part 3)) Use the setup of Exercise 5.
What does the comparison say about the income feature?
Solution 7. On these five training points, knowing the income removes 88% of the squared error. Both numbers are computed on the training points, so the comparison is not a promise about new neighbourhoods, for the reason A First Model gives.