Introduction to Statistical Learning, 2nd ed.
Last modified: 2026-09-29
The Advertising data set records sales of a single product (units in thousands) alongside three media-spend predictors (in thousands of dollars). We’ll use it to make every concept concrete.
One quantitative predictor \(X\), one quantitative response \(Y\):
\[Y = \beta_0 + \beta_1 X + \varepsilon\]
Goal: estimate \(\beta_0, \beta_1\) from a sample and reason about their uncertainty.
Pick \((\hat\beta_0, \hat\beta_1)\) to minimise the residual sum of squares:
\[\text{RSS} = \sum_{i=1}^n \bigl(y_i - \hat\beta_0 - \hat\beta_1 x_i\bigr)^2\]
Closed-form solutions:
\[\hat\beta_1 = \frac{\sum (x_i - \bar x)(y_i - \bar y)}{\sum (x_i - \bar x)^2}, \qquad \hat\beta_0 = \bar y - \hat\beta_1 \bar x\]
sales ~ TVEach \(\hat\beta_j\) is a random variable. Its standard error decreases with sample size and with predictor spread; it increases with residual variance:
\[\widehat{\text{SE}}(\hat\beta_1) = \sqrt{\dfrac{\hat\sigma^2}{\sum (x_i - \bar x)^2}},\qquad \hat\sigma^2 = \frac{\text{RSS}}{n - 2}\]
The \((1 - \alpha)\) confidence interval is
\[\hat\beta_1 \pm t_{n-2, \, 1 - \alpha/2} \cdot \widehat{\text{SE}}(\hat\beta_1).\]
Under \(H_0:\beta_1 = 0\), the studentised slope follows a \(t_{n-2}\) distribution:
\[t = \frac{\hat\beta_1}{\widehat{\text{SE}}(\hat\beta_1)} \;\sim\; t_{n-2}\]
A small \(p\)-value is evidence that TV spend and sales are associated.
Residual standard error measures typical deviation between \(Y\) and \(\hat Y\):
\[\text{RSE} = \sqrt{\text{RSS} / (n - 2)}\]
\(R^2\) is the fraction of variability in \(Y\) explained by the model:
\[R^2 = 1 - \frac{\text{RSS}}{\text{TSS}}, \qquad \text{TSS} = \sum (y_i - \bar y)^2\]
In simple linear regression \(R^2\) equals the squared correlation \(\text{Cor}(X, Y)^2\).
With \(p\) predictors:
\[Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p + \varepsilon\]
Coefficient \(\beta_j\) is the expected change in \(Y\) per unit change in \(X_j\), holding all other predictors fixed. That conditional-on-the-others interpretation is the whole reason to use a multiple regression rather than fitting each predictor alone.
sales ~ TV + radio + newspaperStack the \(p + 1\) coefficients into \(\boldsymbol{\beta}\), the design matrix into \(\mathbf X\) (with a column of ones for the intercept), and the responses into \(\mathbf y\):
\[\mathbf y = \mathbf X \boldsymbol{\beta} + \boldsymbol{\varepsilon}\]
The OLS estimator solves the normal equations \(\mathbf X^\top \mathbf X \, \hat{\boldsymbol{\beta}} = \mathbf X^\top \mathbf y\):
\[\hat{\boldsymbol{\beta}} = (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf y\]
assuming \(\mathbf X^\top \mathbf X\) is invertible (no perfect collinearity).
Test \(H_0:\beta_1 = \cdots = \beta_p = 0\) with the F-statistic:
\[F = \frac{(\text{TSS} - \text{RSS})/p}{\text{RSS}/(n - p - 1)} \;\sim\; F_{p, \, n - p - 1}\]
A large \(F\) (and small \(p\)-value) says at least one predictor is associated with \(Y\). The individual \(t\)-tests on each \(\hat\beta_j\) then ask which ones.
With \(p\) candidates there are \(2^p\) subsets — too many to try when \(p\) grows. Practical strategies for small to moderate \(p\):
For large \(p\), use the shrinkage and dimension-reduction methods of Chapter 6 (ridge, lasso, PCR, PLS) instead.
For a binary predictor (e.g., own/rent) use one dummy:
\[x_i = \begin{cases} 1 & \text{own} \\ 0 & \text{rent} \end{cases}\]
so \(\beta_0\) is the rent-group mean and \(\beta_0 + \beta_1\) is the own-group mean. The choice of reference level affects the sign of \(\hat\beta_1\) but not its \(p\)-value or the fitted values.
For a factor with \(k\) levels create \(k - 1\) dummies, leaving one as the reference. Each coefficient is the contrast against that baseline.
If the effect of \(X_1\) on \(Y\) depends on the value of \(X_2\), add a product term:
\[Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \beta_3 X_1 X_2 + \varepsilon\]
The conditional slope of \(Y\) in \(X_1\) becomes \(\beta_1 + \beta_3 X_2\) — it shifts with \(X_2\). Hierarchy principle: keep the main effects whenever an interaction is in the model, even if their individual \(p\)-values are large.
In both languages TV * radio expands to TV + radio + TV:radio.
A simple way to fit a non-linear function of one predictor is to add powers of it:
\[Y = \beta_0 + \beta_1 X + \beta_2 X^2 + \cdots + \beta_d X^d + \varepsilon\]
This is still linear in the parameters, so standard OLS theory applies. (More flexible non-linear approaches come in Chapter 7.)
The straight-line assumption is wrong when a residuals-vs-fitted plot shows a systematic curve. Fixes: transform a predictor (log, square root), add a polynomial term, or move to a non-linear method.
If the residual spread grows or shrinks with the fitted value, OLS standard errors are wrong (typically too small). Common remedies: log-transform \(Y\), use weighted least squares, or use a sandwich (HC) standard-error estimator.
Time-ordered or clustered data often have correlated residuals (today’s error predicts tomorrow’s). Confidence intervals shrink artificially. Address with specific models for autocorrelation (e.g. ARIMA, GLS with an AR(1) structure) or with cluster-robust standard errors.
An outlier has a large standardised residual — \(|r_i| > 3\) is a common flag. Outliers inflate the RSE (and hence shrink \(R^2\)). Investigate before deleting: a data-entry error is genuine cleanup, a real observation is data.
A high-leverage point has an extreme combination of predictor values (\(X\)-space outlier). Influence on the fit grows with the leverage
\[h_i = \mathbf x_i^\top (\mathbf X^\top \mathbf X)^{-1} \mathbf x_i, \qquad 0 \le h_i \le 1, \quad \bar h = (p + 1)/n.\]
Values much larger than \(\bar h\) deserve a closer look.
When two (or more) predictors carry nearly the same information, \(\mathbf X^\top \mathbf X\) becomes ill-conditioned: coefficients flip sign, standard errors explode, and individual \(t\)-tests become uninformative.
The variance inflation factor
\[\text{VIF}(\hat\beta_j) = \frac{1}{1 - R^2_{X_j \mid X_{-j}}}\]
flags it; rule of thumb is to be wary of \(\text{VIF} > 5\) (or \(> 10\)).
For a test point \(x_0\), find the \(K\) closest training points and average their \(Y\) values:
\[\hat f(x_0) = \frac{1}{K} \sum_{i \in \mathcal N_0(x_0)} y_i\]
Parametric (linear) vs. non-parametric (KNN) trade-off:
KNN’s nearest “neighbours” become far away once \(p\) is large: in \(\mathbb R^p\) the volume of a ball of radius \(r\) shrinks relative to the volume of the predictor space, so to get \(K\) truly local observations you need a sample size that grows exponentially in \(p\). Linear regression sidesteps the issue by imposing structure.