Introduction to Statistical Learning, 2nd ed.
Last modified: 2026-09-29
| Data set | Input \(X\) | Output \(Y\) | Goal |
|---|---|---|---|
| Wage | age, year, education | wage | predict / understand wage drivers |
| Stock market | yesterday’s returns, volume | up / down | classify tomorrow’s direction |
| NCI60 gene expression | 6,830 gene measurements | cancer type | group cancer types (unsupervised) |
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 3, figsize=(12, 4))
Wage.plot.scatter('age', 'wage', alpha=0.3, ax=axes[0])
Wage.plot.scatter('year', 'wage', alpha=0.3, ax=axes[1])
edu_levels = Wage['education'].cat.categories
axes[2].boxplot(
[Wage.loc[Wage['education'] == e, 'wage'] for e in edu_levels],
tick_labels=edu_levels, # matplotlib ≥3.9; was `labels=` before
)
axes[2].set_xlabel('Education'); axes[2].set_ylabel('Wage')
axes[2].tick_params(axis='x', rotation=45)
plt.tight_layout()We observe a response \(Y\) and \(p\) predictors \(X = (X_1, \ldots, X_p)\).
We assume:
\[Y = f(X) + \varepsilon\]
Statistical learning is a collection of approaches for estimating \(f\).
Prediction — we treat \(f\) as a black box.
\[\hat{Y} = \hat{f}(X)\]
We care about accuracy, not the form of \(\hat{f}\).
Inference — we want to understand how \(Y\) changes with \(X\).
No explicit assumption about the form of \(f\); seek an estimate that fits the data points as closely as possible without being too rough or wiggly.
Advantage — can fit a wider range of shapes for \(f\).
Disadvantage — require many observations; harder to interpret.
More flexible models can fit more complex \(f\), but are:
Less flexible models are more interpretable and often more accurate when the true \(f\) is nearly linear.
Training MSE — computed on data used to fit the model:
\[\text{MSE}_{\text{train}} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{f}(x_i))^2\]
We really care about test MSE — how well does \(\hat{f}\) predict new observations?
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42)
lm.fit(X_train, y_train)
mse_test = mean_squared_error(y_test, lm.predict(X_test))
print(f"Test MSE: {mse_test:.2f}")For any \(x_0\) and any \(\hat{f}\):
\[E\!\left[(y_0 - \hat{f}(x_0))^2\right] = \underbrace{\operatorname{Var}(\hat{f}(x_0))}_{\text{variance}} + \underbrace{\left[\operatorname{Bias}(\hat{f}(x_0))\right]^2}_{\text{bias}^2} + \underbrace{\operatorname{Var}(\varepsilon)}_{\text{irreducible}}\]
As model flexibility increases:
| Variance | Bias | |
|---|---|---|
| Very flexible | High ↑ | Low ↓ |
| Very inflexible | Low ↓ | High ↑ |
The test MSE is minimised at the sweet spot where variance and bias² together are smallest — the bias-variance trade-off.
The response \(Y\) is now qualitative (a class label).
Training error rate:
\[\frac{1}{n}\sum_{i=1}^{n} \mathbf{1}(y_i \neq \hat{y}_i)\]
We care about the test error rate: how often does the classifier misclassify a new observation?
Assigns each observation to the most probable class given the predictor values:
\[\hat{Y} = \arg\max_j \Pr(Y = j \mid X = x_0)\]
The resulting Bayes error rate (analogous to irreducible error) is:
\[1 - E\!\left[\max_j \Pr(Y = j \mid X)\right]\]
In practice the true conditional probabilities \(\Pr(Y = j \mid X)\) are unknown. KNN estimates them from data.
For a test point \(x_0\), identify the \(K\) closest training observations \(\mathcal{N}_0\) and estimate:
\[\Pr(Y = j \mid X = x_0) = \frac{1}{K}\sum_{i \in \mathcal{N}_0}\mathbf{1}(y_i = j)\]
from sklearn.neighbors import KNeighborsClassifier
# Example: predict stock market direction with KNN (K = 3)
X_smarket = Smarket[['Lag1', 'Lag2']]
y_smarket = Smarket['Direction']
X_tr, X_te, y_tr, y_te = train_test_split(
X_smarket, y_smarket, test_size=0.2, random_state=0)
knn = KNeighborsClassifier(n_neighbors=3).fit(X_tr, y_tr)
print("Test accuracy:", knn.score(X_te, y_te).round(3))KNN illustrates the bias-variance trade-off in the classification setting just as smoothing splines do in regression.