Discriminant Analysis

Published

Last modified: 2026-11-19

Show code
# The Python chunks on this page import these packages; reticulate provisions them.
library(reticulate)
library(tidymodels) # nolint: undesirable_function_linter
── Attaching packages ────────────────────────────────────── tidymodels 1.5.0 ──
✔ broom        1.0.13     ✔ recipes      1.4.0 
✔ dials        1.4.5      ✔ rsample      1.3.2 
✔ dplyr        1.2.1      ✔ tailor       0.1.0 
✔ ggplot2      4.0.3      ✔ tidyr        1.3.2 
✔ infer        1.1.0      ✔ tune         2.1.0 
✔ modeldata    1.6.0      ✔ workflows    1.3.0 
✔ parsnip      1.6.1      ✔ workflowsets 1.1.1 
✔ purrr        1.2.2      ✔ yardstick    1.4.0 
── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ──
✖ purrr::discard() masks scales::discard()
✖ dplyr::filter()  masks stats::filter()
✖ dplyr::lag()     masks stats::lag()
✖ recipes::step()  masks stats::step()
Show code
py_packages <- c(
  "numpy", "pandas", "statsmodels", "scipy", "scikit-learn", "matplotlib",
  "plotly"
)
py_require(py_packages)

This material was a section of the logistic regression page. Its sources are credited in that page’s Source note.

The naive Bayes assumption and its Spambase example moved here from the probability page; they are adapted from Brian Hutchinson’s Fall 2024 DATA 471 probability refresher [1].

1 Classifying with Bayes’ rule

A logistic regression model gives the probability of each value of the label given the features. An alternative is to model the distribution of the features within each class, and to compute the posterior probability of the label by Bayes’ rule. Choosing a multivariate normal distribution for the features of each class gives linear and quadratic discriminant analysis. Their decision boundaries are linear or quadratic, and they are fitted by maximizing the joint likelihood of the labels and the features, not the conditional likelihood of the labels. This page writes \(K\) for the number of classes, as James et al. [2] does.

ExrExercise 1 (Classifying with Bayes’ rule)

Exercise 1 (Classifying with Bayes’ rule) A logistic regression model gives \(\Pr[Y = k \mid \vX = \vx]\) directly. Suppose instead that the label has \(K\) classes and that we knew, for each class \(k = 1, \ldots, K\):

  • the class prior \(\pi_k = \Pr[Y = k]\), the probability of class \(k\) before the features are seen;
  • the class-conditional density \(f_k(\vx)\), the conditional density of the features given \(Y = k\).

Use Bayes’ rule to express \(\Pr[\Outvar = k \mid \vX = \vx]\) in terms of \(\pi_k\) and \(f_k\).

Solution 1. Bayes’ rule gives the posterior as the prior times the class density, divided by the marginal density of \(\vX\). The law of total probability writes that marginal density as \(\sumto{l}{K} \pi_l f_l(\vx)\). So, wherever that sum is positive,

\[ \Pr[Y = k \mid \vX = \vx] = \frac{\pi_k f_k(\vx)}{\sumto{l}{K} \pi_l f_l(\vx)}. \]

With two classes, diseased and healthy, and a single binary feature, a positive test result, this formula is the positive predictive value of a diagnostic test: the prior of the diseased class is the prevalence, the probability of a positive test is the sensitivity in the diseased class, and it is one minus the specificity in the healthy class.

ExrExercise 2 (Classifying with Bayes’ rule (part 2))

Exercise 2 (Classifying with Bayes’ rule (part 2)) Use the setup of Exercise 1.

To classify \(\vx\), we select the class with the largest posterior probability. Which factor in your formula can be omitted when comparing classes, and why?

Solution 2. In the posterior formula of Solution 1, the denominator is the same for every \(k\), because it is the marginal density of \(\vX\), with the class summed out. At a point \(\vx\) where that density is positive, dividing every posterior by the same positive number does not change which posterior is largest. So the predicted class is

\[ \hy(\vx) = \argmax_k \; \pi_k f_k(\vx), \]

and it is enough to compare the products \(\pi_k f_k(\vx)\).

ExrExercise 3 (Classifying with Bayes’ rule (part 3))

Exercise 3 (Classifying with Bayes’ rule (part 3)) Use the setup of Exercise 1.

Where would \(\pi_k\) and \(f_k\) come from in practice?

Solution 3. Both are estimated from the training data. The proportion of training points in class \(k\) estimates \(\pi_k\). Estimating \(f_k\) means choosing a family of densities for the features within a class and fitting it to the training points of that class. Each choice of family gives a different classifier. The rest of this section uses the multivariate normal family.

LemTheorem 1 (The posterior from a prior and class densities)

Theorem 1 (The posterior from a prior and class densities) Let class \(k\) have prior \(\pi_k > 0\) and feature density \(f_k(\vx)\), for \(k = 1, \dots, K\), and let \(\vx\) be a point where the marginal density \(\sumto{l}{K} \pi_l f_l(\vx)\) is greater than \(0\). Then

\[ \Pr[Y = k \mid \vX = \vx] = \frac{\pi_k f_k(\vx)}{\sumto{l}{K} \pi_l f_l(\vx)}, \]

and the class with the largest posterior is \(\hy(\vx) = \argmax_k \; \pi_k f_k(\vx)\).

Proof. By Solution 1 and Solution 2.

ExmExample 1 (A posterior from two class densities)

Example 1 (A posterior from two class densities) Two classes have priors \(\pi_1 = 0.3\) and \(\pi_2 = 0.7\), and at an observed \(\vx\) their class densities are \(f_1(\vx) = 0.5\) and \(f_2(\vx) = 0.1\). Find the posteriors and the class with the larger posterior.

Show code
br_prior <- c(0.3, 0.7)
br_dens <- c(0.5, 0.1)
br_joint <- br_prior * br_dens
br_post <- br_joint / sum(br_joint)
rbind(joint = br_joint, posterior = br_post)
               [,1]      [,2]
joint     0.1500000 0.0700000
posterior 0.6818182 0.3181818

The products are 0.15 and 0.07, so by Theorem 1 the posteriors are 0.682 and 0.318. Class 1 has the larger posterior, although class 2 has the larger prior.

Show code
library(ggplot2)
bf_steps <- c("prior", "prior times density", "posterior")
bf <- data.frame(
  step = factor(rep(bf_steps, each = 2), levels = bf_steps),
  class = rep(c("class 1", "class 2"), times = 3),
  value = c(br_prior, br_joint, br_post)
)
ggplot(bf) +
  aes(
    step, value,
    colour = class, shape = class, linetype = class, group = class
  ) +
  geom_line() +
  geom_point(size = 3) +
  scale_colour_manual(values = c("class 1" = "#0072B2", "class 2" = "#D55E00")) +
  labs(x = NULL, y = "Value", colour = NULL, shape = NULL, linetype = NULL)
Show code
import matplotlib.pyplot as plt
import numpy as np

prior_bf = np.array([0.3, 0.7])
dens_bf = np.array([0.5, 0.1])
joint_bf = prior_bf * dens_bf
post_bf = joint_bf / joint_bf.sum()
steps_bf = ["prior", "prior times density", "posterior"]
values_bf = np.vstack([prior_bf, joint_bf, post_bf])
fig, ax = plt.subplots(figsize=(5, 4), layout="constrained")
for k, (label, colour, marker, style) in enumerate(
    [("class 1", "#0072B2", "o", "-"), ("class 2", "#D55E00", "^", "--")]
):
    ax.plot(steps_bf, values_bf[:, k], color=colour, marker=marker, markersize=8,
            linestyle=style, label=label);
ax.set_ylabel("Value");
fig.legend(loc="outside lower center", ncol=2);
plt.show()
Figure 1: The prior, the prior times the density, and the posterior of the two classes of Example 1. Class 2 has the larger prior, but the density of class 1 is large enough at this observation that class 1 has the larger posterior.
(a)
Plot with three steps on the horizontal axis: prior, prior times density, and posterior, and the value on the vertical axis from 0 to 1. Class 1 is drawn as blue circles joined by a solid line, class 2 as vermilion triangles joined by a dashed line. Class 1 goes from 0.3 at the prior, to 0.15 for prior times density, to about 0.68 at the posterior. Class 2 goes from 0.7 at the prior, to 0.07 for prior times density, to about 0.32 at the posterior. The two classes swap order: class 2 is higher at the prior and class 1 is higher at the posterior.
(b)
Plot with three steps on the horizontal axis: prior, prior times density, and posterior, and the value on the vertical axis from 0 to 1. Class 1 is drawn as blue circles joined by a solid line, class 2 as vermilion triangles joined by a dashed line. Class 1 goes from 0.3 at the prior, to 0.15 for prior times density, to about 0.68 at the posterior. Class 2 goes from 0.7 at the prior, to 0.07 for prior times density, to about 0.32 at the posterior. The two classes swap order: class 2 is higher at the prior and class 1 is higher at the posterior.
DefDefinition 1 (Bayes classifier)

Definition 1 (Bayes classifier) Let \(\pi_k\) and \(f_k\) be the true class priors and class-conditional densities of the features. The Bayes classifier assigns each \(\vx\) to the class with the largest posterior probability,

\[ \hy(\vx) = \argmax_k \; \pi_k f_k(\vx), \]

as in Theorem 1. No classifier has a smaller probability of misclassifying a new observation [2, Ch. 2], [3].

ExmExample 2 (The Bayes classifier for two normal classes)

Example 2 (The Bayes classifier for two normal classes) Suppose that a single feature \(X\) is normal with mean \(0\) and variance \(1\) in class \(0\), and with mean \(2\) and variance \(1\) in class \(1\), and that the true priors are \(\pi_0 = 0.6\) and \(\pi_1 = 0.4\). Find the rule of the Bayes classifier and its probability of misclassifying a new observation.

Show code
bc_prior <- c(0.6, 0.4)
bc_mean <- c(0, 2)
# pi_0 f_0(x) = pi_1 f_1(x) where 2 x - 2 = log(pi_0 / pi_1); solve for x
bc_cut <- (2 + log(bc_prior[1] / bc_prior[2])) / 2
bc_error <- bc_prior[1] * pnorm(bc_cut, bc_mean[1], 1, lower.tail = FALSE) +
  bc_prior[2] * pnorm(bc_cut, bc_mean[2], 1)
c(cut = bc_cut, error = bc_error) |> round(4)
   cut  error 
1.2027 0.1538 
Show code
import numpy as np
from scipy.stats import norm

prior_bc = np.array([0.6, 0.4])
mean_bc = np.array([0, 2])
# pi_0 f_0(x) = pi_1 f_1(x) where 2 x - 2 = log(pi_0 / pi_1); solve for x
cut_bc = (2 + np.log(prior_bc[0] / prior_bc[1])) / 2
error_bc = prior_bc[0] * norm.sf(cut_bc, mean_bc[0], 1) + prior_bc[1] * norm.cdf(cut_bc, mean_bc[1], 1)
print("cut:", round(cut_bc, 4), "error:", round(error_bc, 4))
cut: 1.2027 error: 0.1538

The products \(\pi_0 f_0(x)\) and \(\pi_1 f_1(x)\) are equal at \(x =\) 1.203, so the Bayes classifier assigns class 0 to every \(x\) below 1.203 and class 1 to every \(x\) above it. The cut point is above the midpoint \(1\) of the two means, because class 0 has the larger prior. The classifier misclassifies a new observation with probability 0.154, and no other rule has a smaller probability for these two classes. Figure 2 plots the two products: the class-0 curve is higher to the left of the dotted cut line, and the class-1 curve is higher to its right.

Show code
# weight each class density by its prior, on a grid of feature values
bc_x <- seq(-3, 5, length.out = 401)
bc_w0 <- bc_prior[1] * dnorm(bc_x, bc_mean[1], 1)
bc_w1 <- bc_prior[2] * dnorm(bc_x, bc_mean[2], 1)
bc_curves <- tibble::tibble(x = bc_x, `class 0` = bc_w0, `class 1` = bc_w1) |>
  tidyr::pivot_longer(-x, names_to = "class", values_to = "weighted")
bc_colours <- c("class 0" = "#0072B2", "class 1" = "#D55E00")
# the Bayes classifier switches class where the two curves cross
bc_curves |>
  ggplot() +
  aes(x = x, y = weighted, colour = class, linetype = class) +
  geom_line() +
  geom_vline(xintercept = bc_cut, linetype = "dotted") +
  scale_colour_manual(values = bc_colours) +
  labs(x = "x", y = "Prior times density", colour = NULL, linetype = NULL)
Show code
import matplotlib.pyplot as plt

# weight each class density by its prior, on a grid of feature values
x_bc = np.linspace(-3, 5, 401)
w0_bc = prior_bc[0] * norm.pdf(x_bc, mean_bc[0], 1)
w1_bc = prior_bc[1] * norm.pdf(x_bc, mean_bc[1], 1)
fig, ax = plt.subplots(figsize=(5, 3.5), layout="constrained")
ax.plot(x_bc, w0_bc, color="#0072B2", linestyle="-", label="class 0");
ax.plot(x_bc, w1_bc, color="#D55E00", linestyle="--", label="class 1");
# the Bayes classifier switches class where the two curves cross
ax.axvline(cut_bc, color="black", linestyle=":");
ax.set_xlabel("x");
ax.set_ylabel("Prior times density");
ax.legend();
plt.show()
Figure 2: The joint density \(\dens(Y = k,\, X = x)\) of class \(k\) and the feature value \(x\), for each class \(k\) of Example 2, plotted against \(x\). The joint density equals \(\pi_k f_k(x)\), the class’s prior times the class’s density. The dotted line marks where the Bayes classifier switches classes.
(a)
Two bell-shaped curves over x from -3 to 5. The class 0 curve, solid blue, peaks at about 0.24 at x = 0. The class 1 curve, dashed vermilion, peaks at about 0.16 at x = 2. A dotted vertical line at x of about 1.2 marks where the curves cross: the class 0 curve is higher to its left and the class 1 curve is higher to its right.
(b)
Two bell-shaped curves over x from -3 to 5. The class 0 curve, solid blue, peaks at about 0.24 at x = 0. The class 1 curve, dashed vermilion, peaks at about 0.16 at x = 2. A dotted vertical line at x of about 1.2 marks where the curves cross: the class 0 curve is higher to its left and the class 1 curve is higher to its right.
DefDefinition 2 (Generative classifier (plug-in Bayes classifier))

Definition 2 (Generative classifier (plug-in Bayes classifier)) A generative classifier is a classifier built on a generative model. It models the joint distribution of the label and the features through a class prior \(\pi_k\) and a class-conditional density \(f_k(\vx)\) for each class \(k\), estimates both from training data [4], and classifies each \(\vx\) by the largest estimated posterior probability \[ \widehat{\Pr}[\Outvar = k \mid \vX = \vx] = \frac{\hat\pi_k \hf_k(\vx)}{\sumto{l}{K} \hat\pi_l \hf_l(\vx)}, \] the formula of Theorem 1 with the estimates \(\hat\pi_k\) and \(\hf_k\) in place of the true priors and densities. Because it puts estimates into the Bayes classifier, it is also called a plug-in Bayes classifier.

ExmExample 3 (A Gaussian generative classifier for the stock market)

Example 3 (A Gaussian generative classifier for the stock market) Return to the Smarket data of the Smarket fit exercise. Model the previous day’s return, Lag1, within each direction as a Gaussian distribution, fitted to the days before 2005. Then classify a day whose previous return was \(1\%\).

da <- readr::read_csv("data/smarket.csv", show_col_types = FALSE)
da_train <- da[da$Year < 2005, ]

direction_counts <- table(da_train$Direction)
prior <- prop.table(direction_counts)
class_mean <- tapply(da_train$Lag1, da_train$Direction, mean)
class_sd <- tapply(da_train$Lag1, da_train$Direction, sd)

x0 <- 1
joint <- prior * dnorm(x0, class_mean, class_sd)
posterior <- joint / sum(joint)
# nolint start: undesirable_function_linter. A matrix.
gen_table <- cbind(prior, mean = class_mean, sd = class_sd, posterior)
# nolint end
round(gen_table, 4)
     prior    mean     sd posterior
Down 0.492  0.0428 1.2274    0.5059
Up   0.508 -0.0395 1.2317    0.4941
import numpy as np
import pandas as pd
from scipy.stats import norm

da_py = pd.read_csv("data/smarket.csv")
da_train_py = da_py[da_py["Year"] < 2005]

by_dir_py = da_train_py.groupby("Direction")["Lag1"]
prior_py = by_dir_py.size() / len(da_train_py)
mean_py = by_dir_py.mean()
sd_py = by_dir_py.std()

x0_py = 1.0
joint_py = prior_py * norm.pdf(x0_py, mean_py, sd_py)
posterior_py = joint_py / joint_py.sum()
print(pd.DataFrame({"prior": prior_py, "mean": mean_py, "sd": sd_py,
                    "posterior": posterior_py}).round(4))
           prior    mean      sd  posterior
Direction                                  
Down       0.492  0.0428  1.2274     0.5059
Up         0.508 -0.0395  1.2317     0.4941

The classes are nearly balanced, with 50.8% of the training days up. The mean of Lag1 is -0.040 on days that went up and 0.043 on days that went down, and the standard deviations are 1.232 and 1.227, almost equal. A day that follows a \(1\%\) return has posterior probability 0.494 of going up, so the classifier predicts Down, with a posterior close to one half. The two class-conditional densities overlap almost completely, which is the same difficulty that the Smarket probabilities figure shows for logistic regression.

Show code
library(ggplot2)
gcf_grid <- seq(-4, 4, length.out = 241)
gcf_curves <- do.call(rbind, lapply(names(class_mean), function(dir) {
  data.frame(
    lag1 = gcf_grid,
    weighted = prior[[dir]] * dnorm(gcf_grid, class_mean[[dir]], class_sd[[dir]]),
    direction = dir
  )
}))
gcf_curves$direction <- factor(gcf_curves$direction, levels = c("Up", "Down"))
ggplot(gcf_curves) +
  aes(lag1, weighted, colour = direction, linetype = direction) +
  geom_line(linewidth = 0.9) +
  geom_vline(xintercept = x0, linetype = "dotted", colour = "grey40") +
  scale_colour_manual(values = c(Up = "#0072B2", Down = "#D55E00")) +
  scale_linetype_manual(values = c(Up = "solid", Down = "dashed")) +
  labs(
    x = "Previous day's return (percent)", y = "Prior times density",
    colour = "Direction", linetype = "Direction"
  )
Show code
import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import norm

grid_gcf = np.linspace(-4, 4, 241)
fig, ax = plt.subplots(figsize=(6, 4), layout="constrained")
for direction, colour, style in [("Up", "#0072B2", "-"), ("Down", "#D55E00", "--")]:
    ax.plot(grid_gcf, prior_py[direction] * norm.pdf(grid_gcf, mean_py[direction], sd_py[direction]),
            color=colour, linestyle=style, linewidth=2, label=direction);
ax.axvline(x0_py, color="grey", linestyle=":");
ax.set_xlabel("Previous day's return (percent)");
ax.set_ylabel("Prior times density");
fig.legend(title="Direction", loc="outside lower center", ncol=2);
plt.show()
Figure 3: Each class’s prior times its density for the previous day’s return, from Example 3. At a return of \(1\%\) (dotted line) the two curves almost coincide, so the classifier cannot separate the classes.
(a)
Plot of two nearly identical bell-shaped curves over the previous day's return, from -4 to 4 percent, one for days that went up and one for days that went down. Each curve is a class density multiplied by its class prior. The solid blue curve for up days and the dashed vermilion curve for down days almost coincide and both peak near 0 at about 0.16, the up curve very slightly higher. A dotted vertical grey line at a return of 1 percent crosses both curves at almost the same height, about 0.115.
(b)
Plot of two nearly identical bell-shaped curves over the previous day's return, from -4 to 4 percent, one for days that went up and one for days that went down. Each curve is a class density multiplied by its class prior. The solid blue curve for up days and the dashed vermilion curve for down days almost coincide and both peak near 0 at about 0.16, the up curve very slightly higher. A dotted vertical grey line at a return of 1 percent crosses both curves at almost the same height, about 0.115.
RemarkRemark 1 (A class-conditional density can be a PMF)

Remark 1 (A class-conditional density can be a PMF). When the features are discrete, such as counts or qualitative features, the class-conditional density \(f_k(\vx)\) of Definition 2 is the PMF \(\Pr[\vX = \vx \mid Y = k]\) [5, p. 146]. The word “density” still fits: a PMF is a density with respect to counting measure, just as a PDF is a density with respect to Lebesgue measure, so PMFs and PDFs differ only in the reference measure.

RemarkRemark 2 (A discriminative classifier for the Smarket training days)

Remark 2 (A discriminative classifier for the Smarket training days). Logistic regression is a discriminative model: it models \(\Pr[Y = k \mid \vX = \vx]\) directly and does not model the distribution of the features. For comparison with Example 3, the code fits a logistic regression of the direction on Lag1 to the Smarket training days before 2005 and computes the probability of an up day after a \(1\%\) return.

disc_fit <- glm(
  I(Direction == "Up") ~ Lag1,
  family = binomial, data = da_train
)
disc_day <- tibble::tibble(Lag1 = x0)
disc_prob <- predict(disc_fit, disc_day, type = "response")
disc_coef <- coef(disc_fit)
disc_out <- c(disc_coef, p_up = disc_prob[[1]])
round(disc_out, 4)
(Intercept)        Lag1        p_up 
     0.0322     -0.0546      0.4944 
import statsmodels.api as sm
import statsmodels.formula.api as smf

da_train_py = da_train_py.assign(up=(da_train_py["Direction"] == "Up").astype(int))
disc_py = smf.glm("up ~ Lag1", data=da_train_py, family=sm.families.Binomial()).fit()
print(disc_py.params.round(4).to_dict())
{'Intercept': 0.0322, 'Lag1': -0.0546}
print("p_up:", round(float(disc_py.predict(pd.DataFrame({"Lag1": [x0_py]})).iloc[0]), 4))
p_up: 0.4944

The fitted model has an intercept and a slope, and nothing else: it has no prior, no class mean and no class variance, because it does not model how Lag1 is distributed. After a \(1\%\) return it gives an up day probability 0.4944, close to the 0.4941 of the generative classifier.

2 Linear discriminant analysis

ExrExercise 4 (Gaussian classes with one covariance matrix)

Exercise 4 (Gaussian classes with one covariance matrix) Let \(\vX \in \reals^{\ninputvars}\), and suppose that within class \(k\) the features are multivariate normal, \(\vX \mid Y = k \sim \ndist{\vm_k, \Sigma}\), with a mean \(\vm_k\) for each class and one covariance matrix \(\Sigma\) shared by all of them. The density is

\[ f_k(\vx) = \frac{1}{(2\pi)^{\ninputvars/2} \abs{\Sigma}^{1/2}} \expt\paren{-\frac12 \qform{\inv{\Sigma}}{\deviationv_k}}, \]

where \(\deviationv_k \eqdef \devof{\vx}{\vm_k}\) is the deviation of \(\vx\) from the class mean.

Take the logarithm of \(\pi_k f_k(\vx)\), and expand the quadratic form.

Solution 4. Assume that \(\Sigma\) is positive definite and every \(\pi_k > 0\). Then \(\inv{\Sigma}\) exists, because a positive definite matrix is invertible, and \(\abs{\Sigma} > 0\), so \(\log\abs{\Sigma}\) is defined. Every normal density is positive, so every product \(\pi_k f_k(\vx)\) is positive and has a logarithm. Write \(q_k(\vx) = \tp{(\vx - \vm_k)} \inv{\Sigma} (\vx - \vm_k)\) for the quadratic form. Taking the logarithm of the product turns it into a sum:

\[ \begin{aligned} \log\paren{\pi_k f_k(\vx)} &= \log \pi_k + \log f_k(\vx) && \text{(log of a product)} \\ &= \log \pi_k + \log\paren{\frac{1}{(2\pi)^{\ninputvars/2} \abs{\Sigma}^{1/2}} \expt\paren{-\frac12 q_k(\vx)}} && \text{(the Gaussian density)} \\ &= \log \pi_k + \log\frac{1}{(2\pi)^{\ninputvars/2} \abs{\Sigma}^{1/2}} + \log \expt\paren{-\frac12 q_k(\vx)} && \text{(log of a product)} \\ &= \log \pi_k - \log\paren{(2\pi)^{\ninputvars/2} \abs{\Sigma}^{1/2}} + \log \expt\paren{-\frac12 q_k(\vx)} && \text{(log of a reciprocal)} \\ &= \log \pi_k - \log\paren{(2\pi)^{\ninputvars/2} \abs{\Sigma}^{1/2}} - \frac12 q_k(\vx) && \text{(log of an exponential)} \\ &= \log \pi_k - \log (2\pi)^{\ninputvars/2} - \log \abs{\Sigma}^{1/2} - \frac12 q_k(\vx) && \text{(log of a product)} \\ &= \log \pi_k - \frac{\ninputvars}{2}\log(2\pi) - \frac12 \log\abs{\Sigma} - \frac12 q_k(\vx) && \text{(log of a power)}. \end{aligned} \]

Next, expand the quadratic form. Because \(\inv{\Sigma}\) is symmetric, the two cross terms are equal: \(\tp{\vm_k}\inv{\Sigma}\vx = \tp{\vx}\inv{\Sigma}\vm_k\).

\[ \begin{aligned} q_k(\vx) &= \tp{\vx}\inv{\Sigma}\vx - \tp{\vx}\inv{\Sigma}\vm_k - \tp{\vm_k}\inv{\Sigma}\vx + \tp{\vm_k}\inv{\Sigma}\vm_k && \text{(distribute)} \\ &= \tp{\vx}\inv{\Sigma}\vx - 2\,\tp{\vx}\inv{\Sigma}\vm_k + \tp{\vm_k}\inv{\Sigma}\vm_k && \text{(symmetry of $\inv{\Sigma}$)}. \end{aligned} \]

Substituting the expansion into the logarithm gives

\[ \begin{aligned} \log\paren{\pi_k f_k(\vx)} &= \log \pi_k - \frac{\ninputvars}{2}\log(2\pi) - \frac12 \log\abs{\Sigma} - \frac12 \paren{\tp{\vx}\inv{\Sigma}\vx - 2\,\tp{\vx}\inv{\Sigma}\vm_k + \tp{\vm_k}\inv{\Sigma}\vm_k} && \text{(substitute $q_k(\vx)$)} \\ &= \log \pi_k - \frac{\ninputvars}{2}\log(2\pi) - \frac12 \log\abs{\Sigma} - \frac12 \tp{\vx} \inv{\Sigma} \vx + \tp{\vx} \inv{\Sigma} \vm_k - \frac12 \tp{\vm_k} \inv{\Sigma} \vm_k && \text{(distribute $-\tfrac12$)}. \end{aligned} \]

ExrExercise 5 (Gaussian classes with one covariance matrix (part 2))

Exercise 5 (Gaussian classes with one covariance matrix (part 2)) Use the setup of Exercise 4.

Drop every term that is the same for all classes, and denote the remaining expression by \(\delta_k(\vx)\). Why does dropping those terms preserve \(\argmax_k \pi_k f_k(\vx)\)?

Solution 5. Three terms, \(-\frac{\ninputvars}{2}\log(2\pi)\), \(-\frac12 \log\abs{\Sigma}\) and \(-\frac12 \tp{\vx} \inv{\Sigma} \vx\), do not depend on \(k\). Adding the same number to every class’s score does not change which score is largest, so they can be dropped. The score that remains is

\[ \delta_k(\vx) = \tp{\vx} \inv{\Sigma} \vm_k - \frac12 \tp{\vm_k} \inv{\Sigma} \vm_k + \log \pi_k. \]

ExrExercise 6 (Gaussian classes with one covariance matrix (part 3))

Exercise 6 (Gaussian classes with one covariance matrix (part 3)) Use the setup of Exercise 4.

Is \(\delta_k(\vx)\) an affine function of \(\vx\), a linear function plus a constant?

Solution 6. Yes: \(\delta_k(\vx)\) is an affine function of \(\vx\). The quadratic term \(\tp{\vx}\inv{\Sigma}\vx\) is the same in every class and was dropped, which is possible only because the covariance matrix is shared. What remains is the dot product of \(\vx\) with the fixed vector \(\inv{\Sigma}\vm_k\), plus the term \(-\frac12 \tp{\vm_k} \inv{\Sigma} \vm_k + \log \pi_k\), which does not depend on \(\vx\). The “linear” in linear discriminant analysis refers to this affine form, whose decision boundaries are hyperplanes.

DefDefinition 3 (Linear discriminant function)

Definition 3 (Linear discriminant function) For class \(k\) with prior \(\pi_k\) and mean \(\vm_k\), and a covariance matrix \(\Sigma\) shared by all classes, the linear discriminant function is

\[ \delta_k(\vx) = \tp{\vx} \inv{\Sigma} \vm_k - \frac12 \tp{\vm_k} \inv{\Sigma} \vm_k + \log \pi_k. \]

ExmExample 4 (The linear discriminant function with one feature)

Example 4 (The linear discriminant function with one feature) Let \(\ninputvars = 1\), with class means \(\vm_0 = 0\) and \(\vm_1 = 2\), shared variance \(\Sigma = 1\), and priors \(\pi_0 = 0.6\) and \(\pi_1 = 0.4\). Evaluate the linear discriminant function of each class at \(\vx = 1.2\).

Show code
ldf_m <- c(0, 2)
ldf_pi <- c(0.6, 0.4)
ldf_x <- 1.2
ldf_delta <- ldf_x * ldf_m - ldf_m^2 / 2 + log(ldf_pi)
ldf_delta
[1] -0.5108256 -0.5162907

With one feature and \(\Sigma = 1\), \(\delta_k(\vx) = \vx \vm_k - \vm_k^2 / 2 + \log \pi_k\), so \(\delta_0(1.2) =\) -0.511 and \(\delta_1(1.2) =\) -0.516.

LemTheorem 2 (The linear discriminant function)

Theorem 2 (The linear discriminant function) Suppose that within class \(k\) the \(\ninputvars\) features are multivariate Gaussian, \(\vX \mid Y = k \sim \ndist{\vm_k, \Sigma}\), with one positive definite covariance matrix \(\Sigma\) shared by all classes, and prior \(\pi_k > 0\) for every class. Then the marginal density of \(\vX\) is positive everywhere:

\[ \pf{\vx} > 0 \quad \text{for every } \vx \in \reals^{\ninputvars}. \tag{1}\]

The class that maximizes \(\pi_k f_k(\vx)\) also maximizes the linear discriminant function \(\delta_k(\vx)\):

\[ \argmax_k \pi_k f_k(\vx) = \argmax_k \delta_k(\vx). \tag{2}\]

Each \(\delta_k\) is an affine function of \(\vx\). More precisely, \(\log\paren{\pi_k f_k(\vx)} = \delta_k(\vx) + c(\vx)\), where \(c(\vx) = -\frac{\ninputvars}{2}\log(2\pi) - \frac12 \log\abs{\Sigma} - \frac12 \tp{\vx} \inv{\Sigma} \vx\) is the same for every class.

ExmExample 5 (The linear discriminant with one feature)

Example 5 (The linear discriminant with one feature) Let \(\ninputvars = 1\), with class means \(\vm_0 = 0\) and \(\vm_1 = 2\), shared variance \(\Sigma = 1\), and priors \(\pi_0 = 0.6\) and \(\pi_1 = 0.4\). Classify the point \(\vx = 1.2\).

Show code
ld_m <- c(0, 2)
ld_pi <- c(0.6, 0.4)
ld_x <- 1.2
ld_delta <- ld_x * ld_m - ld_m^2 / 2 + log(ld_pi)
ld_joint <- ld_pi * dnorm(ld_x, ld_m, 1)
# nolint start: undesirable_function_linter. A matrix.
rbind(delta = ld_delta, pi_times_f = ld_joint)
                 [,1]       [,2]
delta      -0.5108256 -0.5162907
pi_times_f  0.1165116  0.1158766
Show code
# nolint end

The discriminants are -0.511 and -0.516, so Theorem 2 predicts class 0. The products \(\pi_k f_k(\vx)\), 0.1165 and 0.1159, give the same class, 0. The point is close to the boundary: the two discriminants differ by only 0.0055.

Show code
library(ggplot2)
ldf_grid <- seq(-2, 4, length.out = 121)
ldf_lines <- do.call(rbind, lapply(1:2, function(k) {
  data.frame(
    x = ldf_grid,
    delta = ldf_grid * ld_m[k] - ld_m[k]^2 / 2 + log(ld_pi[k]),
    class = paste("class", k - 1)
  )
}))
ggplot(ldf_lines) +
  aes(x, delta, colour = class, linetype = class) +
  geom_line(linewidth = 0.9) +
  geom_vline(xintercept = ld_x, linetype = "dotted", colour = "grey40") +
  scale_colour_manual(values = c("class 0" = "#0072B2", "class 1" = "#D55E00")) +
  scale_linetype_manual(values = c("class 0" = "solid", "class 1" = "dashed")) +
  labs(x = "Feature x", y = "Discriminant", colour = NULL, linetype = NULL)
Show code
import matplotlib.pyplot as plt
import numpy as np

grid_ldf = np.linspace(-2, 4, 121)
fig, ax = plt.subplots(figsize=(6, 4), layout="constrained")
for k, colour, style in [(0, "#0072B2", "-"), (1, "#D55E00", "--")]:
    ax.plot(grid_ldf, grid_ldf * r.ld_m[k] - r.ld_m[k] ** 2 / 2 + np.log(r.ld_pi[k]),
            color=colour, linestyle=style, linewidth=2, label=f"class {k}");
ax.axvline(r.ld_x, color="grey", linestyle=":");
ax.set_xlabel("Feature x");
ax.set_ylabel("Discriminant");
fig.legend(loc="outside lower center", ncol=2);
plt.show()
Figure 4: The linear discriminant functions of class \(0\) and class \(1\) plot as two straight-line graphs against the feature \(x\), and the point \(x = 1.2\) (dotted vertical line) is assigned to the class with the higher discriminant there (Example 5).
(a)
Plot of the two discriminant functions of class 0 and class 1 against the feature x, from -2 to 4. The solid blue line for class 0 is flat at about -0.51. A dashed vermilion line for class 1 rises with slope 2 from about -6.9 at x of -2 to about 5.1 at x of 4, and crosses the class 0 line close to x of 1.2. A dotted vertical grey line at x of 1.2 shows that the two lines are almost equal there, with the class 0 line very slightly higher.
(b)
Plot of the two discriminant functions of class 0 and class 1 against the feature x, from -2 to 4. The solid blue line for class 0 is flat at about -0.51. A dashed vermilion line for class 1 rises with slope 2 from about -6.9 at x of -2 to about 5.1 at x of 4, and crosses the class 0 line close to x of 1.2. A dotted vertical grey line at x of 1.2 shows that the two lines are almost equal there, with the class 0 line very slightly higher.
DefDefinition 4 (Linear discriminant analysis (LDA))

Definition 4 (Linear discriminant analysis (LDA)) Linear discriminant analysis is the generative classifier with a multivariate normal density in each class and one covariance matrix \(\Sigma\) shared by all classes [6]. It predicts the class with the largest linear discriminant function of Theorem 2, with each parameter replaced by its estimate from \(n\) training points, \(n_k\) of them in class \(k\):

  • \(\hat\pi_k = n_k / n\), the class proportion;
  • \(\hat{\vm}_k\), the average of the feature vectors in class \(k\);
  • \(\hSigma = \frac{1}{n - K}\sumkK \sum_{i : y_i = k} \soprod{\deviationv_i}\), the pooled within-class covariance matrix, where \(\deviationv_i \eqdef \devof{\vx_i}{\hat{\vm}_{y_i}}\) is the deviation of training point \(i\) from the average of its class.
ExmExample 6 (LDA for one predictor)

Example 6 (LDA for one predictor) Fit LDA to the Smarket days before 2005 with Lag1 as the only predictor, so \(\ninputvars = 1\) and \(\Sigma\) is a single variance. Compute the pooled variance and the discriminant functions at \(x_0 = 1\) by hand, and compare the resulting posterior probabilities with a library’s.

n_k <- table(da_train$Direction)
pooled <- sum((n_k - 1) * class_sd^2) / (sum(n_k) - 2)
delta <- x0 * class_mean / pooled - class_mean^2 / (2 * pooled) + log(prior)
exp_delta <- exp(delta)
by_hand <- exp_delta / sum(exp_delta)

lda_fit <- MASS::lda(Direction ~ Lag1, data = da_train)
lda_day <- tibble::tibble(Lag1 = x0)
library_post <- predict(lda_fit, lda_day)$posterior
pooled_shown <- round(pooled, 4)
cat("pooled variance:", pooled_shown, "\n")
pooled variance: 1.5119 
# nolint start: undesirable_function_linter. A matrix.
post_table <- rbind(by_hand = by_hand, library = library_post[1, ])
# nolint end
round(post_table, 4)
          Down     Up
by_hand 0.5056 0.4944
library 0.5056 0.4944
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

n_k_py = by_dir_py.size()
pooled_py = ((n_k_py - 1) * sd_py**2).sum() / (n_k_py.sum() - 2)
delta_py = x0_py * mean_py / pooled_py - mean_py**2 / (2 * pooled_py) + np.log(prior_py)
by_hand_py = np.exp(delta_py) / np.exp(delta_py).sum()

lda_py = LinearDiscriminantAnalysis().fit(da_train_py[["Lag1"]], da_train_py["Direction"])
library_py = lda_py.predict_proba(pd.DataFrame({"Lag1": [x0_py]}))[0]
print("pooled variance:", round(float(pooled_py), 4))
pooled variance: 1.5119
print("by hand:", by_hand_py.round(4).to_dict())
by hand: {'Down': 0.5056, 'Up': 0.4944}
print("library:", dict(zip(lda_py.classes_.tolist(), library_py.round(4).tolist())))
library: {'Down': 0.5056, 'Up': 0.4944}

The pooled variance is 1.512, between the two class variances 1.507 and 1.517. The two discriminant functions differ by only 0.022 at \(x_0\). The hand computation and the library give the same posteriors to four decimal places in both languages. With almost a thousand training days, dividing the pooled sum by \(n\) instead of \(n - K\) changes them only in the fifth decimal place. The posterior probability of Up is 0.4944, close to the 0.4941 of Example 3. The small difference from that example’s posterior comes from replacing the two class variances by their pool.

RemarkRemark 3 (The name “discriminant analysis”)

Remark 3 (The name “discriminant analysis”). Despite its name, discriminant analysis is a generative method, not a discriminative one. It models the distribution of the features within each class, \(f_k(\vx)\), and the class proportions \(\pi_k\), and classifies by Bayes’ rule. Logistic regression, in contrast, models \(\Pr[Y = k \mid \vX = \vx]\) directly. The book places discriminant analysis in its section on generative models for classification [5, Sec. 4.4]. The “discriminant” in the name refers to the discriminant functions \(\delta_k(\vx)\) of Theorem 2, which the classifier compares to choose a class, a use of the word that goes back to Fisher’s linear discriminant.

RemarkRemark 4 (The divisor of the pooled covariance matrix)

Remark 4 (The divisor of the pooled covariance matrix). The divisor \(n - K\) in Definition 4 makes \(\hSigma\) an unbiased estimator of \(\Sigma\). Within class \(k\), the sum of the outer products of the deviations from \(\hat{\vm}_k\) has expectation \((n_k - 1)\Sigma\), because estimating the class mean uses up one degree of freedom. Summing over the \(K\) classes gives expectation \((n - K)\Sigma\). The maximum likelihood estimator divides the same sum by \(n\) instead. Software differs: R’s MASS::lda() divides by \(n - K\) [7, Ch. 12]. scikit-learn’s LinearDiscriminantAnalysis depends on its solver and version. Its lsqr and eigen solvers divide by \(n\) in versions 1.8.0 and 1.9.1. Its default svd solver divides by \(n - K\) in version 1.8.0 and by \(n\) in version 1.9.1. With hundreds of training points the two choices give nearly the same classifier.

RemarkRemark 5 (Fisher’s linear discriminant)

Remark 5 (Fisher’s linear discriminant). “Fisher’s linear discriminant” names a closely related method that finds the direction that best separates the class means, relative to the spread within the classes. In the two-class case its direction is proportional to the LDA coefficient vector \(\inv{\Sigma}(\vm_1 - \vm_0)\) [8, Sec. 4.3.3].

ExrExercise 7 (The log odds of LDA)

Exercise 7 (The log odds of LDA) Consider two classes, \(Y \in \set{0, 1}\), under the LDA model. Compare with the log-odds exercise, where the log odds were assumed linear.

Express \(\log \frac{\Pr[\Outvar = 1 \mid \vX = \vx]}{\Pr[\Outvar = 0 \mid \vX = \vx]}\) in terms of the linear discriminant functions \(\delta_1\) and \(\delta_0\).

Solution 7. Both posteriors share the denominator \(\sum_l \pi_l f_l(\vx)\) of Theorem 1, which cancels in their ratio. By Theorem 2, \(\log\paren{\pi_k f_k(\vx)} = \delta_k(\vx) + c(\vx)\), where \(c(\vx)\) collects the terms that the theorem drops and is the same for both classes. So

\[ \begin{aligned} \log \frac{\Pr[Y = 1 \mid \vX = \vx]}{\Pr[Y = 0 \mid \vX = \vx]} &= \log \frac{\pi_1 f_1(\vx)}{\pi_0 f_0(\vx)} && \text{(the denominators cancel)} \\ &= \log\paren{\pi_1 f_1(\vx)} - \log\paren{\pi_0 f_0(\vx)} && \text{(log of a quotient)} \\ &= \paren{\delta_1(\vx) + c(\vx)} - \paren{\delta_0(\vx) + c(\vx)} && \text{(the theorem, for each class)} \\ &= \delta_1(\vx) - \delta_0(\vx) && \text{($c(\vx)$ cancels)}. \end{aligned} \]

ExrExercise 8 (The log odds of LDA (part 2))

Exercise 8 (The log odds of LDA (part 2)) Use the setup of Exercise 7.

Show that the log odds equal \(\coef{0} + \dprod{\vcoef}{\vx}\), and give \(\coef{0}\) and \(\vcoef\) in terms of the parameters of the LDA model.

Solution 8. Subtract the two discriminant functions of Theorem 2:

\[ \begin{aligned} \delta_1(\vx) - \delta_0(\vx) &= \paren{\tp{\vx} \inv{\Sigma} \vm_1 - \frac12 \tp{\vm_1} \inv{\Sigma} \vm_1 + \log \pi_1} \\ &\qquad - \paren{\tp{\vx} \inv{\Sigma} \vm_0 - \frac12 \tp{\vm_0} \inv{\Sigma} \vm_0 + \log \pi_0} && \text{(definition)} \\ &= \tp{\vx} \inv{\Sigma} \vm_1 - \frac12 \tp{\vm_1} \inv{\Sigma} \vm_1 + \log \pi_1 \\ &\qquad - \tp{\vx} \inv{\Sigma} \vm_0 + \frac12 \tp{\vm_0} \inv{\Sigma} \vm_0 - \log \pi_0 && \text{(distribute $-$)} \\ &= \tp{\vx} \inv{\Sigma} \vm_1 - \tp{\vx} \inv{\Sigma} \vm_0 - \frac12 \paren{\tp{\vm_1} \inv{\Sigma} \vm_1 - \tp{\vm_0} \inv{\Sigma} \vm_0} + \log \pi_1 - \log \pi_0 && \text{(regroup)} \\ &= \tp{\vx} \inv{\Sigma} (\vm_1 - \vm_0) - \frac12 \paren{\tp{\vm_1} \inv{\Sigma} \vm_1 - \tp{\vm_0} \inv{\Sigma} \vm_0} + \log \pi_1 - \log \pi_0 && \text{(factor out $\tp{\vx} \inv{\Sigma}$)} \\ &= \tp{\vx} \inv{\Sigma} (\vm_1 - \vm_0) - \frac12 \paren{\tp{\vm_1} \inv{\Sigma} \vm_1 - \tp{\vm_0} \inv{\Sigma} \vm_0} + \log \frac{\pi_1}{\pi_0} && \text{(log of a quotient)} \\ &= \tp{\vx} \inv{\Sigma} (\vm_1 - \vm_0) - \frac12 \tp{(\vm_1 + \vm_0)} \inv{\Sigma} (\vm_1 - \vm_0) + \log \frac{\pi_1}{\pi_0} && \text{(symmetry of $\inv{\Sigma}$)}. \end{aligned} \]

The last step uses the symmetry of \(\inv{\Sigma}\): expanding \(\tp{(\vm_1 + \vm_0)} \inv{\Sigma} (\vm_1 - \vm_0)\) gives four terms, and the cross terms \(-\tp{\vm_1}\inv{\Sigma}\vm_0\) and \(\tp{\vm_0}\inv{\Sigma}\vm_1\) are equal and opposite, so they cancel. Because \(\inv{\Sigma}\) is symmetric, \(\tp{\vx} \inv{\Sigma} (\vm_1 - \vm_0)\) is the dot product of \(\vx\) with \(\inv{\Sigma}(\vm_1 - \vm_0)\). So the log odds have the form \(\coef{0} + \dprod{\vcoef}{\vx}\), with

\[ \vcoef = \inv{\Sigma}(\vm_1 - \vm_0), \qquad \coef{0} = \log \frac{\pi_1}{\pi_0} - \frac12 \tp{(\vm_1 + \vm_0)} \vcoef. \]

ExrExercise 9 (The log odds of LDA (part 3))

Exercise 9 (The log odds of LDA (part 3)) Use the setup of Exercise 7.

What form does \(\Pr[\Outvar = 1 \mid \vX = \vx]\) take under the LDA model?

Solution 9. The sigmoid-from-log-odds theorem turns a linear log odds into the probability, which gives

\[\Pr[Y = 1 \mid \vX = \vx] = \sigmoidf{\coef{0} + \dprod{\vcoef}{\vx}}.\]

This equation is the logistic regression model, with these particular coefficients.

LemTheorem 3 (LDA implies a logistic model)

Theorem 3 (LDA implies a logistic model) Under the two-class LDA model, with \(\Sigma\) positive definite and \(\pi_0, \pi_1 > 0\),

\[ \Pr[Y = 1 \mid \vX = \vx] = \sigmoidf{\coef{0} + \dprod{\vcoef}{\vx}}, \qquad \vcoef = \inv{\Sigma}(\vm_1 - \vm_0), \quad \coef{0} = \log \frac{\pi_1}{\pi_0} - \frac12 \tp{(\vm_1 + \vm_0)} \vcoef. \]

The decision boundary at the threshold \(1/2\), the set where \(\coef{0} + \dprod{\vcoef}{\vx} = 0\), is a hyperplane with normal vector \(\vcoef\) whenever \(\vm_1 \ne \vm_0\), by the decision-boundary theorem.

ExmExample 7 (LDA and logistic regression on the same days)

Example 7 (LDA and logistic regression on the same days) For the one-predictor LDA fit of Example 6, compute \(\coef{0}\) and \(\coef{1}\) from Theorem 3, and compare them with the coefficients of a logistic regression of Up on Lag1 fitted to the same days.

b1 <- (class_mean[["Up"]] - class_mean[["Down"]]) / pooled
mid <- (class_mean[["Up"]] + class_mean[["Down"]]) / 2
b0 <- log(prior[["Up"]] / prior[["Down"]]) - mid * b1

logit_fit <- glm(
  I(Direction == "Up") ~ Lag1,
  family = binomial, data = da_train
)
lda_coef <- c(b0, b1)
logit_coef <- coef(logit_fit)
# nolint start: undesirable_function_linter. A matrix.
rbind(LDA = lda_coef, logistic = logit_coef) |> round(4)
         (Intercept)    Lag1
LDA           0.0322 -0.0545
logistic      0.0322 -0.0546
# nolint end
import statsmodels.formula.api as smf
import statsmodels.api as sm

b1_py = (mean_py["Up"] - mean_py["Down"]) / pooled_py
mid_py = (mean_py["Up"] + mean_py["Down"]) / 2
b0_py = np.log(prior_py["Up"] / prior_py["Down"]) - mid_py * b1_py

da_train_py = da_train_py.assign(up=(da_train_py["Direction"] == "Up").astype(int))
logit_py = smf.glm("up ~ Lag1", data=da_train_py, family=sm.families.Binomial()).fit()
print("LDA:     ", round(float(b0_py), 4), round(float(b1_py), 4))
LDA:      0.0322 -0.0545
print("logistic:", logit_py.params.round(4).tolist())
logistic: [0.0322, -0.0546]

The two fitted log-odds lines nearly coincide. LDA gives a slope of -0.0545 and logistic regression -0.0546, a difference of 0.3% of the logistic slope, and the intercepts are 0.0322 and 0.0322. Both slopes are negative and small: a rise of one percentage point in the previous day’s return lowers the log odds of an up day by about 0.05. When the class densities are normal with a shared variance, which is the model LDA assumes, the two methods’ estimates converge to the same coefficients as the sample grows [8, Sec. 4.4.5]. Their estimates can differ more when that model fails, for example for a strongly skewed predictor.

Show code
library(ggplot2)
llo_grid <- seq(-5, 5, length.out = 201)
llo_curves <- data.frame(
  lag1 = rep(llo_grid, 2),
  prob = c(plogis(b0 + b1 * llo_grid), plogis(coef(logit_fit)[[1]] + coef(logit_fit)[[2]] * llo_grid)),
  model = rep(c("LDA", "logistic regression"), each = length(llo_grid))
)
llo_curves$model <- factor(llo_curves$model, levels = c("logistic regression", "LDA"))
ggplot(llo_curves) +
  aes(lag1, prob, colour = model, linetype = model) +
  geom_line(linewidth = 0.9) +
  scale_colour_manual(values = c("logistic regression" = "#0072B2", LDA = "#D55E00")) +
  scale_linetype_manual(values = c("logistic regression" = "solid", LDA = "dashed")) +
  labs(
    x = "Previous day's return (percent)", y = "Probability the market goes up",
    colour = NULL, linetype = NULL
  )
Show code
import matplotlib.pyplot as plt
import numpy as np
from scipy.special import expit

grid_llo = np.linspace(-5, 5, 201)
fig, ax = plt.subplots(figsize=(6, 4), layout="constrained")
ax.plot(grid_llo, expit(logit_py.params.iloc[0] + logit_py.params.iloc[1] * grid_llo),
        color="#0072B2", linestyle="-", linewidth=2, label="logistic regression");
ax.plot(grid_llo, expit(b0_py + b1_py * grid_llo),
        color="#D55E00", linestyle="--", linewidth=2, label="LDA");
ax.set_xlabel("Previous day's return (percent)");
ax.set_ylabel("Probability the market goes up");
fig.legend(loc="outside lower center", ncol=2);
plt.show()
Figure 5: The fitted probability that the market goes up, from the two fits of Example 7. The two models give almost the same curve.
(a)
Plot of the probability that the market goes up against the previous day's return, from -5 to 5 percent, for two fitted models. A solid blue curve for logistic regression and a dashed vermilion curve for linear discriminant analysis almost coincide. Both fall slowly from about 0.575 at a return of -5 to about 0.44 at a return of 5, and both pass close to 0.51 at a return of 0.
(b)
Plot of the probability that the market goes up against the previous day's return, from -5 to 5 percent, for two fitted models. A solid blue curve for logistic regression and a dashed vermilion curve for linear discriminant analysis almost coincide. Both fall slowly from about 0.575 at a return of -5 to about 0.44 at a return of 5, and both pass close to 0.51 at a return of 0.
DatasetDataset 1: The credit-card default (Default) data
Dataset 1: The credit-card default (Default) data

Simulated records of 10000 credit-card holders, each saying whether the card holder defaulted on their debt, make up the Default data [2, Sec. 4.1]. The records describe no actual card holders. They are in data/default.csv, one line per card holder, copied from ISLR2, the R package of An Introduction to Statistical Learning [2]. Each line gives:

  • default: the outcome, Yes for a card holder who defaulted and No otherwise.
  • student: Yes for a student and No otherwise.
  • balance: the average balance left on the card after the monthly payment, in dollars.
  • income: the card holder’s income, in dollars.
Code: read the Default data and show it
default_preview <- readr::read_csv("data/default.csv", show_col_types = FALSE)
default_preview
Code: summarize each variable of the Default data
dim(default_preview)
[1] 10000     4
Code: summarize each variable of the Default data
skimr::skim_without_charts(default_preview)

Variable type: character

skim_variable n_missing complete_rate min max empty n_unique whitespace
default 0 1 2 3 0 2 0
student 0 1 2 3 0 2 0

Variable type: numeric

skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100
balance 0 1 835.37 483.71 0.00 481.73 823.64 1166.31 2654.32
income 0 1 33516.98 13336.64 771.97 21340.46 34552.64 43807.73 73554.23
Code: read the Default data and show it
import pandas as pd

pd.set_option("display.max_columns", None)
pd.set_option("display.width", 120)

default_preview = pd.read_csv("data/default.csv")
default_preview
     default student      balance        income
0         No      No   729.526495  44361.625074
1         No     Yes   817.180407  12106.134700
2         No      No  1073.549164  31767.138947
3         No      No   529.250605  35704.493935
4         No      No   785.655883  38463.495879
...      ...     ...          ...           ...
9995      No      No   711.555020  52992.378914
9996      No      No   757.962918  19660.721768
9997      No      No   845.411989  58636.156984
9998      No      No  1569.009053  36669.112365
9999      No     Yes   200.922183  16862.952321

[10000 rows x 4 columns]
Code: summarize each variable of the Default data
default_preview.info()
<class 'pandas.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 4 columns):
 #   Column   Non-Null Count  Dtype  
---  ------   --------------  -----  
 0   default  10000 non-null  str    
 1   student  10000 non-null  str    
 2   balance  10000 non-null  float64
 3   income   10000 non-null  float64
dtypes: float64(2), str(2)
memory usage: 312.6 KB
Code: summarize each variable of the Default data
print(default_preview.describe())
            balance        income
count  10000.000000  10000.000000
mean     835.374886  33516.981876
std      483.714985  13336.639563
min        0.000000    771.967729
25%      481.731105  21340.462903
50%      823.636973  34552.644802
75%     1166.308386  43807.729272
max     2654.322576  73554.233495
Code: summarize each variable of the Default data
print(default_preview.isna().sum())
default    0
student    0
balance    0
income     0
dtype: int64
ExmExample 8 (From class densities to a logistic curve)

Example 8 (From class densities to a logistic curve) Fit the one-predictor LDA model to the Default data, with balance as the predictor and a default (Yes) as class 1. Compute each class’s prior times its normal density over a grid of balances, the posterior probability of default from them by Theorem 1, and the logistic curve of Theorem 3.

dflt <- readr::read_csv("data/default.csv", show_col_types = FALSE)
dflt_stats <- dflt |>
  dplyr::group_by(default) |>
  dplyr::summarise(n = dplyr::n(), mean = mean(balance), var = var(balance))
dflt_n <- sum(dflt_stats$n)
dflt_prior <- dflt_stats$n / dflt_n
names(dflt_prior) <- dflt_stats$default
dflt_mean <- dflt_stats$mean
names(dflt_mean) <- dflt_stats$default
dflt_class_sd <- sqrt(dflt_stats$var)
names(dflt_class_sd) <- dflt_stats$default
dflt_ss <- (dflt_stats$n - 1) * dflt_stats$var
dflt_pooled <- sum(dflt_ss) / (dflt_n - 2)
dflt_sd <- sqrt(dflt_pooled)

dflt_b1 <- (dflt_mean[["Yes"]] - dflt_mean[["No"]]) / dflt_pooled
dflt_mid <- (dflt_mean[["Yes"]] + dflt_mean[["No"]]) / 2
dflt_log_prior_ratio <- log(dflt_prior[["Yes"]] / dflt_prior[["No"]])
dflt_b0 <- dflt_log_prior_ratio - dflt_mid * dflt_b1
dflt_boundary <- -dflt_b0 / dflt_b1

dflt_grid <- seq(0, 2700, by = 10)
# Prior times density, on the log scale, for each class.
dflt_log_yes <- log(dflt_prior[["Yes"]]) +
  dnorm(dflt_grid, dflt_mean[["Yes"]], dflt_sd, log = TRUE)
dflt_log_no <- log(dflt_prior[["No"]]) +
  dnorm(dflt_grid, dflt_mean[["No"]], dflt_sd, log = TRUE)
# Bayes' rule: class 1's share of the two products.
dflt_bayes <- 1 / (1 + exp(dflt_log_no - dflt_log_yes))
dflt_sigmoid <- plogis(dflt_b0 + dflt_b1 * dflt_grid)

dflt_glm <- glm(I(default == "Yes") ~ balance, family = binomial, data = dflt)
dflt_logit <- coef(dflt_glm)
dflt_abs_gap <- abs(dflt_bayes - dflt_sigmoid)
dflt_gap <- max(dflt_abs_gap)
dflt_coef <- c(LDA = dflt_b1, logistic = dflt_logit[["balance"]])
signif(dflt_coef, 3)
     LDA logistic 
  0.0046   0.0055 
import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf
from scipy.stats import norm
from scipy.special import expit

dflt_py = pd.read_csv("data/default.csv")
by_class_py = dflt_py.groupby("default")["balance"]
n_dflt_py = by_class_py.size()
prior_dflt_py = n_dflt_py / n_dflt_py.sum()
mean_dflt_py = by_class_py.mean()
pooled_dflt_py = ((n_dflt_py - 1) * by_class_py.var()).sum() / (n_dflt_py.sum() - 2)
sd_dflt_py = np.sqrt(pooled_dflt_py)

b1_dflt_py = (mean_dflt_py["Yes"] - mean_dflt_py["No"]) / pooled_dflt_py
mid_dflt_py = (mean_dflt_py["Yes"] + mean_dflt_py["No"]) / 2
b0_dflt_py = np.log(prior_dflt_py["Yes"] / prior_dflt_py["No"]) - mid_dflt_py * b1_dflt_py

grid_dflt_py = np.arange(0, 2701, 10)
# Prior times density, on the log scale, for each class.
log_yes_py = np.log(prior_dflt_py["Yes"]) + norm.logpdf(grid_dflt_py, mean_dflt_py["Yes"], sd_dflt_py)
log_no_py = np.log(prior_dflt_py["No"]) + norm.logpdf(grid_dflt_py, mean_dflt_py["No"], sd_dflt_py)
# Bayes' rule: class 1's share of the two products.
bayes_dflt_py = 1 / (1 + np.exp(log_no_py - log_yes_py))
sigmoid_dflt_py = expit(b0_dflt_py + b1_dflt_py * grid_dflt_py)

dflt_py = dflt_py.assign(yes=(dflt_py["default"] == "Yes").astype(int))
glm_dflt_py = smf.glm("yes ~ balance", data=dflt_py, family=sm.families.Binomial()).fit()
print("LDA slope:     ", float(np.round(b1_dflt_py, 6)))
LDA slope:      0.004597
print("logistic slope:", float(np.round(glm_dflt_py.params["balance"], 6)))
logistic slope: 0.005499

In Figure 6, the logarithm of the prior of each class times its density is a parabola in the balance \(\predvar\). The two parabolas have the same \(\predvar^2\) term, \(-\predvar^2 / (2 \Sigma)\), because the two classes share one variance \(\Sigma\), the single entry of the shared covariance matrix of Definition 4, which is \(1 \times 1\) here. Their difference, the log odds of default, is therefore an affine function of the balance, so the posterior from Bayes’ rule is the logistic curve of Theorem 3. On the grid the two curves differ by at most 9.4e-16, the size of floating-point rounding. The LDA coefficient of balance is 0.0046 per dollar, and the posterior crosses one half at a balance of 2009 dollars. The small prior of default, 0.033, moves that crossing well above 1276 dollars, the midpoint of the two class means, where the two densities alone would cross. A logistic regression fitted to the same card holders has the coefficient 0.0055, so its curve is steeper than LDA’s. The two fits differ because the balances do not follow LDA’s model: the sample standard deviations of balance are 341 dollars for the card holders who defaulted and 456 dollars for the others, not one shared value.

Show code
dflt_series <- c(
  "default: log prior times density",
  "no default: log prior times density",
  "posterior by Bayes' rule",
  "logistic curve of the LDA coefficients",
  "logistic regression"
)
dflt_panels <- c("log of prior times density", "probability of default")
dflt_logit_eta <- dflt_logit[[1]] + dflt_logit[[2]] * dflt_grid
dflt_logit_curve <- plogis(dflt_logit_eta)
dflt_n_grid <- length(dflt_grid)
dflt_values <- c(
  dflt_log_yes, dflt_log_no, dflt_bayes, dflt_sigmoid, dflt_logit_curve
)
dflt_panel_of <- dflt_panels[c(1, 1, 2, 2, 2)]
dflt_balance_col <- rep(dflt_grid, 5)
dflt_series_col <- rep(dflt_series, each = dflt_n_grid)
dflt_panel_col <- rep(dflt_panel_of, each = dflt_n_grid)
dflt_curves <- tibble::tibble(
  balance = dflt_balance_col,
  value = dflt_values,
  series = dflt_series_col,
  panel = dflt_panel_col
)
dflt_curves$series <- factor(dflt_curves$series, levels = dflt_series)
dflt_bins <- dflt |>
  dplyr::mutate(band = floor(balance / 200) * 200 + 100) |>
  dplyr::group_by(band) |>
  dplyr::summarise(value = mean(default == "Yes"), .groups = "drop") |>
  dplyr::mutate(panel = dflt_panels[[2]])
dflt_colours <- c("#0072B2", "#D55E00", "black", "#56B4E9", "#E69F00")
names(dflt_colours) <- dflt_series
dflt_types <- c("solid", "dashed", "solid", "dashed", "dotted")
names(dflt_types) <- dflt_series
dflt_bin_aes <- ggplot2::aes(x = band, y = value)
dflt_facets <- ggplot2::vars(panel)
dflt_legend <- ggplot2::guide_legend(ncol = 1)
dflt_curves |>
  ggplot2::ggplot() +
  ggplot2::aes(
    x = balance, y = value, colour = series, linetype = series
  ) +
  ggplot2::geom_line(linewidth = 0.9) +
  ggplot2::geom_point(
    data = dflt_bins,
    mapping = dflt_bin_aes,
    inherit.aes = FALSE,
    colour = "grey45"
  ) +
  ggplot2::geom_vline(
    xintercept = dflt_boundary, linetype = "dotted", colour = "grey40"
  ) +
  ggplot2::facet_wrap(dflt_facets, ncol = 1, scales = "free_y") +
  ggplot2::scale_colour_manual(values = dflt_colours) +
  ggplot2::scale_linetype_manual(values = dflt_types) +
  ggplot2::guides(colour = dflt_legend, linetype = dflt_legend) +
  ggplot2::labs(
    x = "Balance (dollars)", y = NULL, colour = NULL, linetype = NULL
  ) +
  ggplot2::theme(legend.position = "bottom")
Show code
import matplotlib.pyplot as plt

boundary_dflt_py = -b0_dflt_py / b1_dflt_py
band_py = np.floor(dflt_py["balance"] / 200) * 200 + 100
rate_py = dflt_py.groupby(band_py)["yes"].mean()
logit_curve_py = expit(glm_dflt_py.params["Intercept"] + glm_dflt_py.params["balance"] * grid_dflt_py)

fig, (top, bottom) = plt.subplots(2, 1, figsize=(6, 6), sharex=True, layout="constrained")
top.plot(grid_dflt_py, log_yes_py, color="#0072B2", linestyle="-", linewidth=2,
         label="default: log prior times density");
top.plot(grid_dflt_py, log_no_py, color="#D55E00", linestyle="--", linewidth=2,
         label="no default: log prior times density");
top.set_ylabel("log of prior times density");
bottom.plot(grid_dflt_py, bayes_dflt_py, color="black", linestyle="-", linewidth=2,
            label="posterior by Bayes' rule");
bottom.plot(grid_dflt_py, sigmoid_dflt_py, color="#56B4E9", linestyle="--", linewidth=2,
            label="logistic curve of the LDA coefficients");
bottom.plot(grid_dflt_py, logit_curve_py, color="#E69F00", linestyle=":", linewidth=2,
            label="logistic regression");
bottom.scatter(rate_py.index, rate_py.values, color="grey", zorder=3);
bottom.set_ylabel("probability of default");
bottom.set_xlabel("Balance (dollars)");
for ax in (top, bottom):
    ax.axvline(boundary_dflt_py, color="grey", linestyle=":");
fig.legend(loc="outside lower center", ncol=1);
plt.show()
Figure 6: LDA fitted to the Default data in Example 8. Top: the logarithm of the prior of each class times its normal density, with the pooled variance. Bottom: the posterior probability of default from Bayes’ rule, the logistic curve of Theorem 3, a fitted logistic regression, and the share of defaults in each 200-dollar band of balance (gray dots). The dotted vertical line marks where the posterior crosses one half.
(a)
Two stacked panels sharing a horizontal axis of card balance, from 0 to 2700 dollars. The top panel shows the logarithm of each class's prior times its normal density as two downward parabolas of the same width: a solid blue one for card holders who defaulted, peaking near 1750 dollars, and a dashed vermilion one for those who did not, peaking higher near 800 dollars. The parabolas cross near 2000 dollars. The bottom panel shows the probability of default: the posterior from Bayes' rule as a solid black S-shaped curve, the logistic curve of the LDA coefficients as a dashed sky-blue curve lying exactly on it, rising through one half near 2000 dollars, a dotted orange curve for logistic regression that rises more steeply, and gray points for the share of defaults among card holders in each 200-dollar band of balance, which follow the logistic regression.
(b)
Two stacked panels sharing a horizontal axis of card balance, from 0 to 2700 dollars. The top panel shows the logarithm of each class's prior times its normal density as two downward parabolas of the same width: a solid blue one for card holders who defaulted, peaking near 1750 dollars, and a dashed vermilion one for those who did not, peaking higher near 800 dollars. The parabolas cross near 2000 dollars. The bottom panel shows the probability of default: the posterior from Bayes' rule as a solid black S-shaped curve, the logistic curve of the LDA coefficients as a dashed sky-blue curve lying exactly on it, rising through one half near 2000 dollars, a dotted orange curve for logistic regression that rises more steeply, and gray points for the share of defaults among card holders in each 200-dollar band of balance, which follow the logistic regression.
ExrExercise 10 (Same form, different fitting criterion)

Exercise 10 (Same form, different fitting criterion) By Theorem 3, two-class LDA and logistic regression give a posterior probability of the same functional form.

Each method chooses its parameters to maximize a likelihood. State the quantity LDA maximizes and the quantity logistic regression maximizes, in terms of \(\pi_k\), \(f_k\) and \(\Pr[\Outvar = 1 \mid \vX = \vx]\).

Solution 10. LDA maximizes the joint likelihood of the labels and the features,

\[\prodin \pi_{y_i} f_{y_i}(\vx_i),\]

and logistic regression maximizes the conditional likelihood of the labels given the features,

\[\prodin \Pr[Y_i = y_i \mid \vX_i = \vx_i].\]

The LDA maximizers have closed forms: the class proportions, class means, and pooled covariance matrix of Definition 4, with the divisor \(n\) in place of \(n - K\) (Remark 4). The logistic estimates need the iterative methods of the logistic regression page, such as the Newton iteration of the Newton’s method exercise. Let \(\pf{\vx} = \sum_l \pi_l f_l(\vx)\) be the marginal density of the features. By Theorem 1, \(\pi_{y_i} f_{y_i}(\vx_i) = \Pr[Y_i = y_i \mid \vX_i = \vx_i]\, \pf{\vx_i}\), so the joint likelihood is the conditional likelihood times \(\prod_i \pf{\vx_i}\). LDA therefore also fits the marginal distribution of the features, which logistic regression does not model.

ExrExercise 11 (Same form, different fitting criterion (part 2))

Exercise 11 (Same form, different fitting criterion (part 2)) Use the setup of Exercise 10.

Does a logistic model with log odds affine in \(\vx\) imply that the classes are Gaussian with a common covariance matrix? Give a counterexample or a proof.

Solution 11. No: logistic log odds that are an affine function of \(\vx\) do not imply Gaussian classes. Consider a single binary feature \(X \in \set{0, 1}\) with \(\Pr[X = 1 \mid Y = k] = q_k\), where \(0 < q_k < 1\). The class densities are \(f_k(x) = q_k^x (1 - q_k)^{1 - x}\) for \(x \in \set{0, 1}\), with logarithms \(\log f_k(x) = x \log q_k + (1 - x) \log(1 - q_k)\). By Theorem 1, the log odds of \(Y = 1\) are

\[ \begin{aligned} \log \frac{\pi_1 f_1(x)}{\pi_0 f_0(x)} &= \log\frac{\pi_1}{\pi_0} + \log f_1(x) - \log f_0(x) && \text{(log of a quotient of products)} \\ &= \log\frac{\pi_1}{\pi_0} + x \log q_1 + (1 - x) \log(1 - q_1) - x \log q_0 - (1 - x) \log(1 - q_0) && \text{(substitute $\log f_k(x)$)} \\ &= \log\frac{\pi_1}{\pi_0} + x \log\frac{q_1}{q_0} + (1 - x)\log\frac{1 - q_1}{1 - q_0} && \text{(log of a quotient)} \\ &= \log\frac{\pi_1}{\pi_0} + \log\frac{1 - q_1}{1 - q_0} + x \paren{\log\frac{q_1}{q_0} - \log\frac{1 - q_1}{1 - q_0}} && \text{(collect the terms in $x$)}, \end{aligned} \]

which is an affine function of \(x\), although \(X\) is not Gaussian. Many other families of class-conditional densities also give log odds that are affine in \(\vx\), so logistic regression makes weaker distributional assumptions than LDA [8, Sec. 4.4.5].

ExrExercise 12 (Same form, different fitting criterion (part 3))

Exercise 12 (Same form, different fitting criterion (part 3)) Use the setup of Exercise 10.

Which method is more affected when the features are not Gaussian, and which uses the data more efficiently when they are?

Solution 12. LDA is more affected when the features are not Gaussian. Its estimates are then fitted to a misspecified density, and its boundary can have a higher error rate than logistic regression’s. LDA uses the data more efficiently when the features are Gaussian: it is then asymptotically more efficient, because it also fits the marginal distribution of the features, which carries information about the parameters. In practice the two methods often give similar boundaries [8, Sec. 4.4.5].

ExrExercise 13 (Same form, different fitting criterion (part 4))

Exercise 13 (Same form, different fitting criterion (part 4)) Use the setup of Exercise 10.

the separation exercise showed that the logistic regression coefficients have no maximum likelihood estimate when the two classes are perfectly separated. What happens to the LDA parameters in that case?

Solution 13. The LDA parameter estimates remain finite. The class proportions, class means, and pooled covariance matrix of Definition 4 are averages of the data, so they are finite for any data. When the pooled covariance matrix is invertible, the coefficients \(\vcoef\) and \(\coef{0}\) of Theorem 3 are finite too. So LDA gives a finite decision boundary when the classes are linearly separable, and the logistic regression coefficients diverge (the separation exercise).

3 Quadratic discriminant analysis

ExrExercise 14 (Gaussian classes with their own covariance matrices)

Exercise 14 (Gaussian classes with their own covariance matrices) Repeat Exercise 4, but let each class have its own covariance matrix: \(\vX \mid Y = k \sim \ndist{\vm_k, \Sigma_k}\).

Which terms of \(\log\paren{\pi_k f_k(\vx)}\) can still be dropped because they are the same for all classes?

Solution 14. Only \(-\frac{\ninputvars}{2}\log(2\pi)\).

Assume that every \(\Sigma_k\) is positive definite and every \(\pi_k > 0\). Then each \(\inv{\Sigma_k}\) exists, because a positive definite matrix is invertible, each \(\log\abs{\Sigma_k}\) and \(\log \pi_k\) is defined, and every product \(\pi_k f_k(\vx)\) is positive. The derivation of Solution 4 then goes through with \(\Sigma_k\) in place of \(\Sigma\). The term \(-\frac12 \log\abs{\Sigma_k}\) now depends on \(k\), and so does the quadratic form \(\qform{\inv{\Sigma_k}}{\deviationv_k}\), where \(\deviationv_k \eqdef \devof{\vx}{\vm_k}\), because \(\Sigma_k\) is inside it. In particular, \(-\frac12\qform{\inv{\Sigma_k}}{\vx}\) differs between classes, so it cannot be dropped.

ExrExercise 15 (Gaussian classes with their own covariance matrices (part 2))

Exercise 15 (Gaussian classes with their own covariance matrices (part 2)) Use the setup of Exercise 14.

Give the remaining discriminant function \(\delta_k(\vx)\).

Solution 15. With the constant dropped,

\[ \delta_k(\vx) = -\frac12 \log\abs{\Sigma_k} - \frac12 \qform{\inv{\Sigma_k}}{\deviationv_k} + \log \pi_k, \]

where \(\deviationv_k \eqdef \devof{\vx}{\vm_k}\) is the deviation of \(\vx\) from the class mean.

ExrExercise 16 (Gaussian classes with their own covariance matrices (part 3))

Exercise 16 (Gaussian classes with their own covariance matrices (part 3)) Use the setup of Exercise 14.

Is \(\delta_k(\vx)\) an affine function of \(\vx\)?

Solution 16. No: \(\delta_k(\vx)\) is not an affine function of \(\vx\). Expanding the quadratic form as in Solution 4 shows that \(\delta_k(\vx)\) contains \(-\frac12 \tp{\vx}\inv{\Sigma_k}\vx\), a quadratic form in \(\vx\) that is not zero for any \(\vx \ne \vzero\), because \(\inv{\Sigma_k}\) is positive definite.

DefDefinition 5 (Quadric surface, quadric)

Definition 5 (Quadric surface, quadric) A quadric surface, or quadric, in \(\reals^{\ninputvars}\) is a set of the form

\[ \setbuild{\vx \in \reals^{\ninputvars}}{\qform{\matr{A}}{\vx} + \dprod{\va}{\vx} + c = 0}, \]

where \(\matr{A}\) is a nonzero \(\ninputvars \times \ninputvars\) symmetric matrix, \(\va \in \reals^{\ninputvars}\), and \(c \in \reals\) [9].

ExmExample 9 (Quadric surfaces in two dimensions and in one)

Example 9 (Quadric surfaces in two dimensions and in one)  

  • With \(\ninputvars = 2\), \(\matr{A} = \diag(1, 1)\), \(\va = \vzero\), and \(c = -1\), the equation is \(x_1^2 + x_2^2 - 1 = 0\), so the quadric surface is the unit circle.
  • With \(\ninputvars = 1\), \(\matr{A}\) is a nonzero number \(a_2\), \(\va\) is a number \(a_1\), and the equation is \(a_2 x^2 + a_1 x + c = 0\), so the quadric surface is the set of real roots of a quadratic equation: at most two numbers.
ExmExample 10 (Three points on a line are not a quadric surface)

Example 10 (Three points on a line are not a quadric surface) With \(\ninputvars = 1\), the set \(\set{-1, 0, 1}\) is not a quadric surface. With \(\ninputvars = 1\), Definition 5 needs a nonzero number \(a_2\) with \(a_2 x^2 + a_1 x + c = 0\) at each of \(x = -1\), \(x = 0\), and \(x = 1\), and a quadratic equation has at most two real roots.

ExrExercise 17 (Gaussian classes with their own covariance matrices (part 4))

Exercise 17 (Gaussian classes with their own covariance matrices (part 4)) Use the setup of Exercise 14.

What geometric surface is the set \(\setbuild{\vx}{\delta_k(\vx) = \delta_l(\vx)}\) where the discriminant functions of two classes agree?

Solution 17. The difference \(\delta_k(\vx) - \delta_l(\vx)\) is a polynomial of degree at most two in the coordinates of \(\vx\), with quadratic part \(-\frac12 \qform{\paren{\inv{\Sigma_k} - \inv{\Sigma_l}}}{\vx}\). The matrix \(-\frac12 \paren{\inv{\Sigma_k} - \inv{\Sigma_l}}\) is symmetric, and it is nonzero when \(\Sigma_k \ne \Sigma_l\), because \(\inv{\Sigma_k} = \inv{\Sigma_l}\) would force \(\Sigma_k = \Sigma_l\). So when \(\Sigma_k \ne \Sigma_l\), the boundary \(\setbuild{\vx}{\delta_k(\vx) = \delta_l(\vx)}\) is a quadric surface. When \(\ninputvars = 2\), it is a conic section: an ellipse, a parabola, a hyperbola, or a degenerate case such as a pair of lines. When \(\Sigma_k = \Sigma_l\), the quadratic terms and the log-determinant terms cancel, and the boundary is a hyperplane, as for LDA.

ExrExercise 18 (Gaussian classes with their own covariance matrices (part 5))

Exercise 18 (Gaussian classes with their own covariance matrices (part 5)) Use the setup of Exercise 14.

How many points can the boundary between two classes contain when \(\ninputvars = 1\) and \(\Sigma_k \ne \Sigma_l\)?

Solution 18. When \(\ninputvars = 1\) and the two variances differ, the condition \(\delta_k(x) = \delta_l(x)\) is a quadratic equation in \(x\) with a nonzero coefficient of \(x^2\), so the boundary consists of at most two points.

DefDefinition 6 (Quadratic discriminant function)

Definition 6 (Quadratic discriminant function) For class \(k\) with prior \(\pi_k\), mean \(\vm_k\) and covariance matrix \(\Sigma_k\), the quadratic discriminant function is

\[ \delta_k(\vx) = -\frac12 \log\abs{\Sigma_k} - \frac12 \qform{\inv{\Sigma_k}}{\deviationv_k} + \log \pi_k, \]

where \(\deviationv_k \eqdef \devof{\vx}{\vm_k}\) is the deviation of \(\vx\) from the class mean.

ExmExample 11 (The quadratic discriminant function with one feature)

Example 11 (The quadratic discriminant function with one feature) Let \(\ninputvars = 1\), with two classes that have the same mean \(\vm_0 = \vm_1 = 0\), variances \(\Sigma_0 = 1\) and \(\Sigma_1 = 4\), and equal priors. Evaluate the quadratic discriminant function of each class at \(\vx = 0.5\).

Show code
qdf_var <- c(1, 4)
qdf_x <- 0.5
qdf_delta <- -log(qdf_var) / 2 - qdf_x^2 / (2 * qdf_var) + log(0.5)
qdf_delta
[1] -0.8181472 -1.4175444

With one feature and \(\vm_k = 0\), \(\delta_k(\vx) = -\tfrac12 \log \Sigma_k - \vx^2 / (2 \Sigma_k) + \log \pi_k\), so \(\delta_0(0.5) =\) -0.818 and \(\delta_1(0.5) =\) -1.418.

LemTheorem 4 (The quadratic discriminant function)

Theorem 4 (The quadratic discriminant function) Suppose that within class \(k\) the features are multivariate Gaussian, \(\vX \mid Y = k \sim \ndist{\vm_k, \Sigma_k}\), with its own positive definite covariance matrix \(\Sigma_k\), and prior \(\pi_k > 0\) for every class. Then the marginal density of \(\vX\) is positive everywhere:

\[ \pf{\vx} > 0 \quad \text{for every } \vx \in \reals^{\ninputvars}. \tag{3}\]

The class that maximizes \(\pi_k f_k(\vx)\) also maximizes the quadratic discriminant function \(\delta_k(\vx)\):

\[ \argmax_k \pi_k f_k(\vx) = \argmax_k \delta_k(\vx). \tag{4}\]

Proof. By Solution 14 and Solution 15.

ExmExample 12 (The quadratic discriminant with one feature)

Example 12 (The quadratic discriminant with one feature) Let \(\ninputvars = 1\), with two classes that have the same mean \(\vm_0 = 0\) and \(\vm_1 = 0\), variances \(\Sigma_0 = 1\) and \(\Sigma_1 = 4\), and equal priors. Classify \(\vx = 0.5\) and \(\vx = 3\).

Show code
qd_var <- c(1, 4)
qd_delta <- function(x) -log(qd_var) / 2 - x^2 / (2 * qd_var) + log(0.5)
qd_x <- c(0.5, 3)
qd_sd <- sqrt(qd_var)
qd_one <- numeric(1)
qd_class_by_delta <- vapply(qd_x, \(x) {
  qd_d <- qd_delta(x)
  which.max(qd_d) - 1
}, qd_one)
qd_class_by_density <- vapply(qd_x, \(x) {
  qd_f <- dnorm(x, 0, qd_sd)
  which.max(0.5 * qd_f) - 1
}, qd_one)
# delta_0(x) = delta_1(x) where x^2 (1/2 - 1/8) = log(4) / 2
qd_root <- sqrt(log(4) / 2 / (1 / 2 - 1 / 8))
rbind(
  x = qd_x,
  class_by_delta = qd_class_by_delta,
  class_by_density = qd_class_by_density
)
                 [,1] [,2]
x                 0.5    3
class_by_delta    0.0    1
class_by_density  0.0    1

By Theorem 4 the discriminants and the densities give the same classes: class 0 for \(\vx = 0.5\) and class 1 for \(\vx = 3\). The narrower class has the larger discriminant near the shared mean, and the wider class has the larger discriminant far from it. So \(\delta_0 = \delta_1\) at two points, \(x = \pm\) 1.36, the roots of a quadratic equation in \(x\). The rule assigns class 0 to the interval between the roots and class 1 to both tails. No linear discriminant can give this rule: with one feature, an affine discriminant difference changes sign at most once, so it splits the line into two pieces, not three.

Show code
library(ggplot2)
qdf_grid <- seq(-5, 5, length.out = 241)
qdf_curves <- data.frame(
  x = rep(qdf_grid, 2),
  delta = c(sapply(qdf_grid, \(x) qd_delta(x)[1]), sapply(qdf_grid, \(x) qd_delta(x)[2])),
  class = rep(c("variance 1", "variance 4"), each = length(qdf_grid))
)
ggplot(qdf_curves) +
  aes(x, delta, colour = class, linetype = class) +
  geom_line(linewidth = 0.9) +
  geom_vline(xintercept = qd_x, linetype = "dotted", colour = "grey40") +
  scale_colour_manual(values = c("variance 1" = "#0072B2", "variance 4" = "#D55E00")) +
  scale_linetype_manual(values = c("variance 1" = "solid", "variance 4" = "dashed")) +
  labs(x = "Feature x", y = "Discriminant", colour = NULL, linetype = NULL)
Show code
import matplotlib.pyplot as plt
import numpy as np

var_qdf = np.asarray(r.qd_var)
grid_qdf = np.linspace(-5, 5, 241)
fig, ax = plt.subplots(figsize=(6, 4), layout="constrained")
for k, colour, style in [(0, "#0072B2", "-"), (1, "#D55E00", "--")]:
    delta_qdf = -np.log(var_qdf[k]) / 2 - grid_qdf ** 2 / (2 * var_qdf[k]) + np.log(0.5)
    ax.plot(grid_qdf, delta_qdf, color=colour, linestyle=style, linewidth=2,
            label=f"variance {var_qdf[k]:.0f}");
for x_qdf in np.atleast_1d(r.qd_x):
    ax.axvline(x_qdf, color="grey", linestyle=":");
ax.set_xlabel("Feature x");
ax.set_ylabel("Discriminant");
fig.legend(loc="outside lower center", ncol=2);
plt.show()
Figure 7: The two quadratic discriminant functions of Example 12, whose classes have the same mean. The point \(x = 0.5\) (dotted vertical) is between the two crossings, where the class of variance 1 has the higher discriminant, and the point \(x = 3\) is outside them, where the class of variance 4 does, so the two points go to different classes.
(a)
Plot of the two quadratic discriminant functions against the feature x, from -5 to 5. The solid blue curve for the class of variance 1 is a downward parabola peaking at about -0.69 at x of 0 and falling steeply, to about -13.2 at x of 5. The dashed vermilion curve for the class of variance 4 is a flatter downward parabola peaking at about -1.39 at x of 0 and falling to about -4.5 at x of 5. The curves cross at two points, about x of -1.4 and x of 1.4, with the blue curve higher between them and the vermilion curve higher outside. Two dotted vertical grey lines mark x of 0.5, inside the crossings, and x of 3, outside them.
(b)
Plot of the two quadratic discriminant functions against the feature x, from -5 to 5. The solid blue curve for the class of variance 1 is a downward parabola peaking at about -0.69 at x of 0 and falling steeply, to about -13.2 at x of 5. The dashed vermilion curve for the class of variance 4 is a flatter downward parabola peaking at about -1.39 at x of 0 and falling to about -4.5 at x of 5. The curves cross at two points, about x of -1.4 and x of 1.4, with the blue curve higher between them and the vermilion curve higher outside. Two dotted vertical grey lines mark x of 0.5, inside the crossings, and x of 3, outside them.
CorCorollary 1 (Quadratic discriminant boundaries are quadric surfaces)

Corollary 1 (Quadratic discriminant boundaries are quadric surfaces) Under the assumptions of Theorem 4, let \(k\) and \(l\) be two classes with \(\Sigma_k \ne \Sigma_l\). Then the set on which their quadratic discriminant functions agree,

\[ \setbuild{\vx \in \reals^{\ninputvars}}{\delta_k(\vx) = \delta_l(\vx)}, \]

is a quadric surface.

Proof. By Solution 17.

ExmExample 13 (A circular quadratic discriminant boundary)

Example 13 (A circular quadratic discriminant boundary) Let \(\ninputvars = 2\), with two classes that have the same mean \(\vm_0 = \vm_1 = \vzero\) and the same prior \(\pi_0 = \pi_1 = \frac12\). Their covariance matrices are \(\Sigma_0 = \diag(1, 1)\) and \(\Sigma_1 = \diag(4, 4)\). Then \(\abs{\Sigma_0} = 1\) and \(\abs{\Sigma_1} = 16\), so the quadratic discriminant functions are

\[ \delta_0(\vx) = -\frac12 \normsq{\vx} + \log \frac12 \]

and

\[ \delta_1(\vx) = -\frac12 \log 16 - \frac18 \normsq{\vx} + \log \frac12. \]

Show code
qb_radius_sq <- log(16) / 2 / (1 / 2 - 1 / 8)
qb_radius_sq
[1] 3.696785

So \(\delta_0(\vx) = \delta_1(\vx)\) where \(\frac38 \normsq{\vx} - \frac12 \log 16 = 0\), which is the circle \(\normsq{\vx} =\) 3.7 around the origin. By Corollary 1 this circle is a quadric surface, with \(\matr{A} = \frac38 \diag(1, 1)\), \(\va = \vzero\), and \(c = -\frac12 \log 16\).

DefDefinition 7 (Quadratic discriminant analysis (QDA))

Definition 7 (Quadratic discriminant analysis (QDA)) Quadratic discriminant analysis is the generative classifier with a multivariate normal density in each class and a separate covariance matrix \(\Sigma_k\) for each class [10]. It predicts the class with the largest quadratic discriminant function of Theorem 4, with each parameter replaced by its estimate. The priors and means are estimated as in Definition 4. Each \(\hSigma_k\) is the sample covariance matrix of the training points in class \(k\) alone, with divisor \(n_k - 1\). R’s MASS::qda() uses that divisor. scikit-learn’s QuadraticDiscriminantAnalysis uses it in version 1.8.0, and it divides by \(n_k\) in version 1.9.1.

The LDA model is the QDA model with the constraint that \(\Sigma_k = \Sigma\) for every class \(k\).

ExmExample 14 (The quadratic boundary for one predictor)

Example 14 (The quadratic boundary for one predictor) Give the two directions of the Smarket days before 2005 their own variances, as in Example 3, and find where the Up and Down discriminant functions of QDA are equal. Setting \(\delta_{\text{Up}}(x) = \delta_{\text{Down}}(x)\) and collecting terms gives a quadratic equation in \(x\).

delta_qda <- function(x, k, mean, sd, prior) {
  -log(sd[[k]]) - (x - mean[[k]])^2 / (2 * sd[[k]]^2) + log(prior[[k]])
}
delta_up <- function(x) delta_qda(x, "Up", class_mean, class_sd, prior)
delta_down <- function(x) delta_qda(x, "Down", class_mean, class_sd, prior)

# delta_Up - delta_Down is quadratic in x; fit its three coefficients exactly
grid <- c(-1, 0, 1)
gap <- delta_up(grid) - delta_down(grid)
roots <- sort(Re(polyroot(coef(lm(gap ~ grid + I(grid^2))))))
cat("roots:", round(roots, 3), "\n")
roots: 0.539 23.441 
for (x in c(-1, 1, 30)) {
  pred_class <- if (delta_up(x) > delta_down(x)) "Up" else "Down"
  cat(x, "->", pred_class, "\n")
}
-1 -> Up 
1 -> Down 
30 -> Up 
def delta_qda_py(x, k):
    return (-np.log(sd_py[k]) - (x - mean_py[k]) ** 2 / (2 * sd_py[k] ** 2)
            + np.log(prior_py[k]))

# delta_Up - delta_Down is quadratic in x; fit its three coefficients exactly
grid_py = np.array([-1.0, 0.0, 1.0])
coef_py = np.polyfit(grid_py, delta_qda_py(grid_py, "Up") - delta_qda_py(grid_py, "Down"), 2)
roots_py = np.sort(np.roots(coef_py))
print("roots:", roots_py.round(3))
roots: [ 0.539 23.441]
for x in [-1.0, 1.0, 30.0]:
    up = delta_qda_py(x, "Up") > delta_qda_py(x, "Down")
    print(x, "->", "Up" if up else "Down")
-1.0 -> Up
1.0 -> Down
30.0 -> Up

The two discriminant functions are equal at \(x =\) 0.54 and \(x =\) 23.44. The first root is in the middle of the data, where the returns range from -4.9 to 5.7. The second root is far outside that range, so within the range of the data the boundary is a single point, as for LDA. The boundary is curved only because the Up variance, 1.517, differs slightly from the Down variance, 1.507. With the variances this close, the coefficient of \(x^2\) is small, and QDA gives almost the same classifier as LDA.

Show code
library(ggplot2)
lq_x <- seq(min(da_train$Lag1), max(da_train$Lag1), length.out = 300)
lq_post <- function(delta_fun) {
  up <- delta_fun(lq_x, "Up")
  down <- delta_fun(lq_x, "Down")
  exp(up) / (exp(up) + exp(down))
}
lq_lda <- function(x, k) x * class_mean[[k]] / pooled - class_mean[[k]]^2 / (2 * pooled) + log(prior[[k]])
lq_qda <- function(x, k) delta_qda(x, k, class_mean, class_sd, prior)
lq <- data.frame(
  lag1 = rep(lq_x, 2),
  posterior = c(lq_post(lq_lda), lq_post(lq_qda)),
  method = rep(c("LDA", "QDA"), each = length(lq_x))
)
ggplot(lq) +
  aes(lag1, posterior, colour = method, linetype = method) +
  geom_line() +
  scale_colour_manual(values = c(LDA = "#0072B2", QDA = "#D55E00")) +
  labs(x = "Lag1", y = "Posterior probability of Up", colour = NULL, linetype = NULL)
Show code
import matplotlib.pyplot as plt
import numpy as np

x_lq = np.linspace(da_train_py["Lag1"].min(), da_train_py["Lag1"].max(), 300)

def post_lq(delta_up, delta_down):
    return np.exp(delta_up) / (np.exp(delta_up) + np.exp(delta_down))

def delta_lda_py(x, k):
    return x * mean_py[k] / pooled_py - mean_py[k] ** 2 / (2 * pooled_py) + np.log(prior_py[k])

fig, ax = plt.subplots(figsize=(5, 4), layout="constrained")
ax.plot(x_lq, post_lq(delta_lda_py(x_lq, "Up"), delta_lda_py(x_lq, "Down")),
        color="#0072B2", linestyle="-", label="LDA");
ax.plot(x_lq, post_lq(delta_qda_py(x_lq, "Up"), delta_qda_py(x_lq, "Down")),
        color="#D55E00", linestyle="--", label="QDA");
ax.set_xlabel("Lag1");
ax.set_ylabel("Posterior probability of Up");
fig.legend(loc="outside lower center", ncol=2);
plt.show()
Figure 8: The posterior probability of Up given Lag1 under LDA (solid) and QDA (dashed) for the Smarket days before 2005, as in Example 6 and Example 14. The two curves nearly coincide, because the two directions have nearly equal spread.
(a)
Line plot of the posterior probability that the market went Up, against the previous day's return Lag1 over the range of the training data, for LDA and QDA. Both curves fall as Lag1 rises, from about 0.58 on the left to about 0.43 on the right, staying near one half. The solid LDA curve is nearly straight and the dashed QDA curve bends slightly away from it, lying above it at both ends.
(b)
Line plot of the posterior probability that the market went Up, against the previous day's return Lag1 over the range of the training data, for LDA and QDA. Both curves fall as Lag1 rises, from about 0.58 on the left to about 0.43 on the right, staying near one half. The solid LDA curve is nearly straight and the dashed QDA curve bends slightly away from it, lying above it at both ends.

4 Naive Bayes and nearest neighbors

The chain rule of probability holds for every joint distribution, and it has too many parameters to use as it stands: the last factor, \(\Pr(X_N \mid X_1, \dots, X_{N-1})\), is a separate distribution for every combination of the earlier variables. A modeling assumption cuts that down. Conditional independence says that, once some variable is known, other variables carry no further information about each other.

DefDefinition 8 (The naive Bayes assumption)

Definition 8 (The naive Bayes assumption) Let \(Y\) be a class label and \(X_1, \dots, X_{\ninputvars}\) the input features. The naive Bayes assumption is that the features are conditionally independent given the class [11]: \[ \Pr(X_1, \dots, X_{\ninputvars} \mid Y) = \prod_{j=1}^{\ninputvars} \Pr(X_j \mid Y). \]

ExmExample 15 (Do free and money appear independently in spam?)

Example 15 (Do free and money appear independently in spam?) In the Spambase data [12], let \(Y\) be the class (spam or not), \(X_1\) be “the email contains the word free”, and \(X_2\) be “the email contains the word money”, as in the chain rule example. Restrict to the spam emails (\(Y = 1\)). Compare the joint probability \(\Pr(X_1 = 1, X_2 = 1 \mid Y = 1)\) with the value \(\Pr(X_1 = 1 \mid Y = 1)\Pr(X_2 = 1 \mid Y = 1)\) that the naive Bayes assumption would give:

import pandas as pd

spambase_nb = pd.read_csv("data/spambase.csv")
spam_nb = spambase_nb[spambase_nb["Class"] == 1]   # keep only the spam emails
p1_nb = (spam_nb["word_freq_free"] > 0).mean()
p2_nb = (spam_nb["word_freq_money"] > 0).mean()
joint_nb = ((spam_nb["word_freq_free"] > 0) & (spam_nb["word_freq_money"] > 0)).mean()
print(joint_nb, p1_nb * p2_nb)
0.2410369553226696 0.20490275387446608
spambase_nb <- readr::read_csv("data/spambase.csv", show_col_types = FALSE)
spam_nb <- dplyr::filter(spambase_nb, Class == 1) # keep only the spam emails
p1_nb <- mean(spam_nb$word_freq_free > 0)
p2_nb <- mean(spam_nb$word_freq_money > 0)
joint_nb <- mean(spam_nb$word_freq_free > 0 & spam_nb$word_freq_money > 0)
c(joint = joint_nb, product = p1_nb * p2_nb)
    joint   product 
0.2410370 0.2049028 

Among spam, 54.6% of emails contain free and 37.6% contain money, so under the assumption 20.5% would contain both. The data show 24.1%. The assumption is wrong here, since the two words tend to appear together. A naive Bayes classifier uses the product anyway, and it still often classifies well.

The assumption is rarely exactly true, which is why it is called naive. Its value is in how few parameters it needs.

ExmExample 16 (Counting parameters)

Example 16 (Counting parameters) Suppose the class \(Y\) takes \(K\) values and each of \(\ninputvars\) features is binary.

Without the naive Bayes assumption, \(\Pr(X_1, \dots, X_{\ninputvars} \mid Y = k)\) is an arbitrary distribution over \(2^{\ninputvars}\) possible values of the binary features, for each class \(k\). It needs \(2^{\ninputvars} - 1\) free numbers per class, since the probabilities must sum to one. With the assumption, each \(\Pr(X_j \mid Y = k)\) is a single Bernoulli probability, so each class needs only \(\ninputvars\) numbers.

Show code
# The number of free parameters in the class-conditional distributions
# of K classes with D binary features, with and without the assumption.
k_classes <- 2
d_features <- c(5, 20, 30)
full <- k_classes * (2^d_features - 1)
naive <- k_classes * d_features
data.frame(
  D = d_features,
  full = format(full, scientific = FALSE),
  naive = naive
)

With 20 binary features and 2 classes, the full model has about 2,097,150 parameters, more than most datasets have observations. The naive Bayes model has 40. The number grows exponentially in \(\ninputvars\) without the assumption, and linearly in \(\ninputvars\) with it.

DefDefinition 9 (Naive Bayes classifier)

Definition 9 (Naive Bayes classifier) A naive Bayes classifier is the generative classifier whose class-conditional density has one factor for each feature, by the naive Bayes assumption,

\[ f_k(\vx) = \prod_{j=1}^{\ninputvars} f_{kj}(x_j), \tag{5}\]

with each one-feature density \(f_{kj}\) estimated from the training points of class \(k\) [5, Sec. 4.4.4], [13, Sec. 10.1], [14]. The naive Bayes classifier predicts the class with the largest \(\hat\pi_k \prod_{j=1}^{\ninputvars} \hf_{kj}(x_j)\).

DefDefinition 10 (Gaussian naive Bayes classifier)

Definition 10 (Gaussian naive Bayes classifier) A Gaussian naive Bayes classifier is the naive Bayes classifier that takes each \(f_{kj}\) to be a Gaussian density with its own mean \(\mean_{kj}\) and variance \(\sigma_{kj}^2\), estimated by the class-\(k\) sample mean and sample variance of feature \(j\).

ExmExample 17 (Gaussian naive Bayes is QDA with diagonal covariance matrices)

Example 17 (Gaussian naive Bayes is QDA with diagonal covariance matrices) Consider a Gaussian naive Bayes classifier with \(\ninputvars\) features. Its class-conditional density multiplies \(\ninputvars\) normal densities with means \(\mean_{kj}\) and variances \(\sigma_{kj}^2\). That multiplication gives the multivariate normal density with mean vector \(\vm_k = (\mean_{k1}, \dots, \mean_{k\ninputvars})\) and diagonal covariance matrix \(\covmat_k = \diagf{\sigma_{k1}^2, \dots, \sigma_{k\ninputvars}^2}\). So the classifier is QDA with each \(\covmat_k\) restricted to be diagonal.

For the Smarket days before 2005, with features Lag1 and Lag2, the ellipses of Figure 9 have axes parallel to the coordinate axes, because each \(\covmat_k\) is diagonal.

Show code
library(dplyr) # nolint: undesirable_function_linter
library(ggplot2) # nolint: undesirable_function_linter
angle <- seq(0, 2 * pi, length.out = 100)
nbc_angles <- tibble(angle = angle)
nbc_class_stats <- da_train |>
  summarize(
    .by = Direction,
    Lag1_mean = mean(Lag1),
    Lag1_sd = sd(Lag1),
    Lag2_mean = mean(Lag2),
    Lag2_sd = sd(Lag2)
  )
nbc_ellipses <- nbc_class_stats |>
  cross_join(nbc_angles) |>
  mutate(
    Lag1 = Lag1_mean + 2 * Lag1_sd * cos(angle),
    Lag2 = Lag2_mean + 2 * Lag2_sd * sin(angle)
  )
nbc_colours <- c(Up = "#0072B2", Down = "#D55E00")
nbc_linetypes <- c(Up = "solid", Down = "dashed")
da_train |>
  ggplot() +
  aes(Lag1, Lag2, colour = Direction, linetype = Direction) +
  geom_point(alpha = 0.3, size = 0.8) +
  geom_path(data = nbc_ellipses, linewidth = 0.9) +
  scale_colour_manual(values = nbc_colours) +
  scale_linetype_manual(values = nbc_linetypes) +
  labs(
    x = "Previous day's return (percent)",
    y = "Return two days before (percent)"
  )
Show code
import matplotlib.pyplot as plt
import numpy as np

angle_nbc = np.linspace(0, 2 * np.pi, 100)
fig, ax = plt.subplots(figsize=(6, 4), layout="constrained")
for direction, colour, style in [("Up", "#0072B2", "-"), ("Down", "#D55E00", "--")]:
    grp = da_train_py[da_train_py["Direction"] == direction]
    ax.scatter(grp["Lag1"], grp["Lag2"], s=6, alpha=0.3, color=colour);
    ax.plot(grp["Lag1"].mean() + 2 * grp["Lag1"].std() * np.cos(angle_nbc),
            grp["Lag2"].mean() + 2 * grp["Lag2"].std() * np.sin(angle_nbc),
            color=colour, linestyle=style, linewidth=2, label=direction);
ax.set_xlabel("Previous day's return (percent)");
ax.set_ylabel("Return two days before (percent)");
fig.legend(title="Direction", loc="outside lower center", ncol=2);
plt.show()
Figure 9: The class-\(k\) Gaussian naive Bayes density contour two standard deviations from the mean, for each of the two classes.
(a)
Scatter plot of the previous day's return, Lag1, on the horizontal axis against the return two days before, Lag2, on the vertical axis, for stock market days before 2005, with points colored by whether the index went up or down that day. The two clouds overlap almost completely. Over the points are two axis-aligned ellipses, a solid blue one for up days and a dashed vermilion one for down days, nearly the same size and centered near the origin.
(b)
Scatter plot of the previous day's return, Lag1, on the horizontal axis against the return two days before, Lag2, on the vertical axis, for stock market days before 2005, with points colored by whether the index went up or down that day. The two clouds overlap almost completely. Over the points are two axis-aligned ellipses, a solid blue one for up days and a dashed vermilion one for down days, nearly the same size and centered near the origin.
DefDefinition 11 (K-nearest neighbors classifier)

Definition 11 (K-nearest neighbors classifier) For a positive integer \(K\) of neighbors and a feature vector \(\vx\), the K-nearest neighbors classifier finds the \(K\) training points closest to \(\vx\) in Euclidean distance, and predicts the most frequent class among them [5, Sec. 4.7.6]. The K-nearest neighbors classifier estimates \(\Pr(\Outvar = k \mid \vX = \vx)\) by the fraction of those \(K\) points in class \(k\), so K-nearest neighbors makes no assumption about the form of \(f_k\). The nonparametric models chapter studies K-nearest neighbors further.

ExmExample 18 (The nearest neighbors of one Smarket day)

Example 18 (The nearest neighbors of one Smarket day) Consider a day on which Lag1 was \(0.5\) and Lag2 was \(-0.5\). For \(K = 5\), the five Smarket training days closest to that point in Euclidean distance are the neighbors the K-nearest neighbors classifier counts.

knn_query <- c(Lag1 = 0.5, Lag2 = -0.5)
knn_diff1 <- da_train$Lag1 - knn_query[["Lag1"]]
knn_diff2 <- da_train$Lag2 - knn_query[["Lag2"]]
knn_dist <- sqrt(knn_diff1^2 + knn_diff2^2)
knn_nearest <- da_train$Direction[order(knn_dist)[1:5]]
table(knn_nearest)
knn_nearest
Down   Up 
   3    2 
import numpy as np

knn_dist_py = np.hypot(r.da_train["Lag1"] - 0.5, r.da_train["Lag2"] + 0.5)
knn_nearest_py = r.da_train["Direction"].iloc[np.argsort(knn_dist_py)[:5]]
knn_nearest_py.value_counts()
Direction
Down    3
Up      2
Name: count, dtype: int64

Of the five neighbors, 2 were up days and 3 were down days, so the classifier estimates \(\Pr(\Outvar = \text{Up} \mid \vX = \vx)\) as 0.4 and predicts Down.

5 Comparing the methods

ExrExercise 19 (Counting parameters)

Exercise 19 (Counting parameters) Consider a classification problem with \(K\) classes and \(\ninputvars\) features. A symmetric \(\ninputvars \times \ninputvars\) matrix has \(\ninputvars(\ninputvars+1)/2\) free entries. Count the free parameters of LDA (Definition 4).

Solution 19. LDA has \(K - 1\) free priors, because the priors sum to one, and \(K\ninputvars\) entries in the \(K\) class means. Its one covariance matrix is symmetric, so it adds \(\ninputvars(\ninputvars+1)/2\) free entries. The total is \((K - 1) + K\ninputvars + \ninputvars(\ninputvars+1)/2\) parameters.

ExrExercise 20 (Counting parameters (part 2))

Exercise 20 (Counting parameters (part 2)) Use the setup of Exercise 19.

Count the free parameters of QDA (Definition 7).

Solution 20. QDA has the same \(K - 1\) priors and \(K\ninputvars\) mean entries as LDA, and \(K\) covariance matrices instead of one: \((K - 1) + K\ninputvars + K\,\ninputvars(\ninputvars+1)/2\) parameters.

ExrExercise 21 (Counting parameters (part 3))

Exercise 21 (Counting parameters (part 3)) Use the setup of Exercise 19.

Count the free parameters of the naive Bayes assumption with a Gaussian density for each feature in each class. This assumption makes each class covariance matrix diagonal.

Solution 21. Gaussian naive Bayes has the same \(K - 1\) priors and \(K\ninputvars\) mean entries, and \(K\) diagonal covariance matrices, which hold \(K\ninputvars\) variances: \((K - 1) + 2K\ninputvars\) parameters.

ExrExercise 22 (Counting parameters (part 4))

Exercise 22 (Counting parameters (part 4)) Use the setup of Exercise 19.

Count the free parameters of multinomial logistic regression with a reference class, as in the second softmax exercise, so that it has \(K - 1\) coefficient vectors, each with an intercept.

Solution 22. Each of the \(K - 1\) coefficient vectors has an intercept and \(\ninputvars\) slopes, so multinomial logistic regression has \((K - 1)(\ninputvars + 1)\) parameters.

ExrExercise 23 (Counting parameters (part 5))

Exercise 23 (Counting parameters (part 5)) Use the setup of Exercise 19.

Which counts grow fastest in \(\ninputvars\)?

Solution 23. As \(\ninputvars\) grows, the LDA count grows like \(\ninputvars^2/2\), and the QDA count like \(K \ninputvars^2/2\). The naive Bayes and logistic counts grow only linearly in \(\ninputvars\). Each QDA covariance matrix is estimated from only the \(n_k\) points of its own class, so when the classes are small, QDA’s estimates vary more from one training set to another than those of any of the other models.

LemTheorem 5 (Parameter counts of the classifiers)

Theorem 5 (Parameter counts of the classifiers) With \(K\) classes and \(\ninputvars\) features, the number of free parameters is

Table 1: The number of free parameters of each method.
Model Parameters
LDA \((K - 1) + K\ninputvars + \ninputvars(\ninputvars+1)/2\)
QDA \((K - 1) + K\ninputvars + K\,\ninputvars(\ninputvars+1)/2\)
Gaussian naive Bayes \((K - 1) + 2K\ninputvars\)
Multinomial logistic regression \((K - 1)(\ninputvars + 1)\)
ExmExample 19 (Parameters for two classes)

Example 19 (Parameters for two classes) Count the parameters of Theorem 5 for \(K = 2\) classes and several numbers of features \(\ninputvars\), including \(\ninputvars = 2\), for two lagged returns, and the \(\ninputvars = 6\) of the Smarket fit exercise.

k_classes <- 2
p_features <- c(2, 6, 50, 100)
counts <- data.frame(
  p = p_features,
  LDA = (k_classes - 1) + k_classes * p_features +
    p_features * (p_features + 1) / 2,
  QDA = (k_classes - 1) + k_classes * p_features +
    k_classes * p_features * (p_features + 1) / 2,
  `Gaussian naive Bayes` = (k_classes - 1) + 2 * k_classes * p_features,
  logistic = (k_classes - 1) * (p_features + 1),
  check.names = FALSE
)
counts
K_py = 2
p_py = np.array([2, 6, 50, 100])
counts_py = pd.DataFrame({
    "p": p_py,
    "LDA": (K_py - 1) + K_py * p_py + p_py * (p_py + 1) // 2,
    "QDA": (K_py - 1) + K_py * p_py + K_py * p_py * (p_py + 1) // 2,
    "Gaussian naive Bayes": (K_py - 1) + 2 * K_py * p_py,
    "logistic": (K_py - 1) * (p_py + 1),
})
print(counts_py.to_string(index=False))
  p  LDA   QDA  Gaussian naive Bayes  logistic
  2    8    11                     9         3
  6   34    55                    25         7
 50 1376  2651                   201        51
100 5251 10301                   401       101

At \(\ninputvars =\) 50 the QDA model has 2,651 parameters and LDA has 1,376, against 51 for logistic regression. The Smarket training set has 998 days, which is 0.38 observations per QDA parameter at \(\ninputvars =\) 50. At \(\ninputvars =\) 100 QDA has 10,301 parameters, more than the 998 training days.

Show code
library(ggplot2)
dpf_long <- reshape(
  counts, direction = "long", idvar = "p",
  varying = list(c("LDA", "QDA", "Gaussian naive Bayes", "logistic")),
  v.names = "parameters", timevar = "method",
  times = c("LDA", "QDA", "Gaussian naive Bayes", "logistic")
)
dpf_long$method <- factor(dpf_long$method, levels = c("QDA", "LDA", "Gaussian naive Bayes", "logistic"))
ggplot(dpf_long) +
  aes(p, parameters, colour = method, linetype = method, shape = method) +
  geom_hline(yintercept = nrow(da_train), linetype = "dotted", colour = "grey40") +
  geom_line(linewidth = 0.9) +
  geom_point(size = 2.5) +
  scale_x_log10() +
  scale_y_log10() +
  scale_colour_manual(values = c(
    QDA = "#D55E00", LDA = "#0072B2", `Gaussian naive Bayes` = "#009E73", logistic = "grey30"
  )) +
  scale_linetype_manual(values = c(
    QDA = "solid", LDA = "dashed", `Gaussian naive Bayes` = "dotdash", logistic = "longdash"
  )) +
  scale_shape_manual(values = c(QDA = 17, LDA = 16, `Gaussian naive Bayes` = 15, logistic = 18)) +
  labs(x = "Number of features", y = "Number of parameters", colour = NULL, linetype = NULL, shape = NULL)
Show code
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(6, 4), layout="constrained")
ax.axhline(len(da_train_py), color="grey", linestyle=":");
for method, colour, style, marker in [
    ("QDA", "#D55E00", "-", "^"), ("LDA", "#0072B2", "--", "o"),
    ("Gaussian naive Bayes", "#009E73", "-.", "s"), ("logistic", "#4d4d4d", (0, (8, 3)), "D"),
]:
    ax.plot(counts_py["p"], counts_py[method], color=colour, linestyle=style, marker=marker,
            label=method);
ax.set_xscale("log");
ax.set_yscale("log");
ax.set_xlabel("Number of features");
ax.set_ylabel("Number of parameters");
fig.legend(loc="outside lower center", ncol=2);
plt.show()
Figure 10: The parameter counts of Example 19 for two classes, with the number of Smarket training days (dotted line). The quadratic model has more parameters than there are training days somewhere between 6 and 50 features.
(a)
Plot of the number of parameters of four classifiers for two classes against the number of features, 2, 6, 50 and 100, on logarithmic axes. The curve for quadratic discriminant analysis rises most steeply, from 11 parameters at 2 features to about 10,300 at 100. The curve for linear discriminant analysis rises more slowly, to about 5,250, the Gaussian naive Bayes curve to about 400, and the logistic regression curve lowest, to 101. A dotted horizontal grey line marks the number of training days, 998, which the quadratic curve crosses between 6 and 50 features.
(b)
Plot of the number of parameters of four classifiers for two classes against the number of features, 2, 6, 50 and 100, on logarithmic axes. The curve for quadratic discriminant analysis rises most steeply, from 11 parameters at 2 features to about 10,300 at 100. The curve for linear discriminant analysis rises more slowly, to about 5,250, the Gaussian naive Bayes curve to about 400, and the logistic regression curve lowest, to 101. A dotted horizontal grey line marks the number of training days, 998, which the quadratic curve crosses between 6 and 50 features.
ExrExercise 24 (Choosing among the classifiers)

Exercise 24 (Choosing among the classifiers) LDA, QDA, Gaussian naive Bayes, and logistic regression are all linear or quadratic classifiers fitted to the same data. For each situation in this exercise and the next three, state which you would try first, and why.

There are \(\nobs = 60\) observations in two classes and \(\ninputvars = 40\) features.

Solution 24. Gaussian naive Bayes or a regularized version of logistic regression, not QDA, and LDA only with caution.

ExrExercise 25 (Choosing among the classifiers (part 2))

Exercise 25 (Choosing among the classifiers (part 2)) Use the setup of Exercise 24.

There are \(\nobs = 100{,}000\) observations and \(\ninputvars = 3\) features, and a plot shows that the within-class spread of the features differs a lot between classes.

Solution 25. QDA. With \(3\) features and two classes, QDA has 19 parameters and LDA 13, by Theorem 5, and they are estimated from \(100{,}000\) points. The unequal spreads are structure that LDA’s single covariance matrix cannot represent, so LDA is biased here, and QDA can represent it.

ExrExercise 26 (Choosing among the classifiers (part 3))

Exercise 26 (Choosing among the classifiers (part 3)) Use the setup of Exercise 24.

Some features are binary indicators, such as whether a word appears in an email.

Solution 26. Not LDA or QDA. They model the features with a Gaussian density, which is a poor model of a binary feature. Logistic regression does not model the distribution of the features, so binary features need no density, and naive Bayes can use a Bernoulli density for each one, as in the parameter count for binary features.

ExrExercise 27 (Choosing among the classifiers (part 4))

Exercise 27 (Choosing among the classifiers (part 4)) Use the setup of Exercise 24.

How is the choice a bias-variance trade-off?

Solution 27. Moving from LDA to QDA adds parameters. That lowers the bias when the classes really differ in their covariance matrices, and it raises the variance of the estimates, because each class covariance matrix is estimated from that class alone. LDA accepts bias from forcing one covariance matrix on all classes, in exchange for estimating only one matrix, from all \(\nobs\) points. Gaussian naive Bayes restricts each class covariance matrix to be diagonal instead, so it has the fewest parameters of the three generative models when \(\ninputvars\) is large, and it is biased when the features are correlated within a class. QDA is the better choice when \(\nobs\) is large relative to its number of parameters in Theorem 5, and LDA when \(\nobs\) is small [2, Ch. 4]. Regularized discriminant analysis compromises between the two by shrinking each \(\hSigma_k\) toward the pooled \(\hSigma\) [8, Sec. 4.3.1].

6 Lab: discriminant analysis, naive Bayes and nearest neighbors on the stock market data

ExrExercise 28 (LDA and QDA on the stock market)

Exercise 28 (LDA and QDA on the stock market) Use the Smarket days of the Smarket fit exercise. Fit LDA and QDA for the direction of the market with Lag1 and Lag2 as the only predictors, on the days before 2005. Predict each day of 2005 and tabulate the predictions against what happened.

What are the fitted class proportions and class means?

Solution 28.

da_test <- da[da$Year == 2005, ]
lda2 <- MASS::lda(Direction ~ Lag1 + Lag2, data = da_train)
qda2 <- MASS::qda(Direction ~ Lag1 + Lag2, data = da_train)
# The six-predictor logistic regression of the Smarket fit exercise, scored on the same days.
glm_fit <- glm(
  I(Direction == "Up") ~ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume,
  data = da_train,
  family = binomial
)
pred_glm <- ifelse(predict(glm_fit, da_test, type = "response") > 0.5, "Up", "Down")
acc_glm <- mean(pred_glm == da_test$Direction)

lda2$prior
    Down       Up 
0.491984 0.508016 
lda2$means
            Lag1        Lag2
Down  0.04279022  0.03389409
Up   -0.03954635 -0.03132544
pred_lda <- predict(lda2, da_test)$class
pred_qda <- predict(qda2, da_test)$class
table(LDA = pred_lda, actual = da_test$Direction)
      actual
LDA    Down  Up
  Down   35  35
  Up     76 106
table(QDA = pred_qda, actual = da_test$Direction)
      actual
QDA    Down  Up
  Down   30  20
  Up     81 121
acc <- c(
  LDA = mean(pred_lda == da_test$Direction),
  QDA = mean(pred_qda == da_test$Direction),
  always_up = mean(da_test$Direction == "Up")
)
round(acc, 3)
      LDA       QDA always_up 
    0.560     0.599     0.560 
# 95% intervals for each accuracy, and for the paired differences between rules
# scored on the same days, as for the logistic regression of the Smarket fit exercise.
wald_ci <- function(p, n) {
  p + c(lower = -1, upper = 1) * qnorm(0.975) * sqrt(p * (1 - p) / n)
}
paired_ci <- function(a, b) {
  d <- as.integer(a) - as.integer(b)
  mean(d) + c(lower = -1, upper = 1) * qt(0.975, length(d) - 1) * sd(d) / sqrt(length(d))
}
ci_acc <- sapply(acc, wald_ci, n = nrow(da_test))
hit_lda <- pred_lda == da_test$Direction
hit_qda <- pred_qda == da_test$Direction
hit_up <- da_test$Direction == "Up"
ci_qda_up <- paired_ci(hit_qda, hit_up)
ci_qda_lda <- paired_ci(hit_qda, hit_lda)
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis

features_py = ["Lag1", "Lag2"]
da_test_py = da_py[da_py["Year"] == 2005]
lda2_py = LinearDiscriminantAnalysis().fit(da_train_py[features_py], da_train_py["Direction"])
qda2_py = QuadraticDiscriminantAnalysis().fit(da_train_py[features_py], da_train_py["Direction"])

print("priors:", dict(zip(lda2_py.classes_.tolist(), lda2_py.priors_.round(3).tolist())))
priors: {'Down': 0.492, 'Up': 0.508}
print(pd.DataFrame(lda2_py.means_, index=lda2_py.classes_, columns=features_py).round(3))
       Lag1   Lag2
Down  0.043  0.034
Up   -0.040 -0.031
for name, model in [("LDA", lda2_py), ("QDA", qda2_py)]:
    pred = model.predict(da_test_py[features_py])
    acc_py = float((pred == da_test_py["Direction"]).mean())
    # A 95% interval for the accuracy: the proportion plus or minus 1.96 standard errors.
    half_py = 1.96 * np.sqrt(acc_py * (1 - acc_py) / len(da_test_py))
    print(name, "accuracy:", round(acc_py, 3), [round(float(acc_py - half_py), 3), round(float(acc_py + half_py), 3)])
    print(pd.crosstab(pred, da_test_py["Direction"], rownames=["predicted"], colnames=["actual"]))
LDA accuracy: 0.56 [0.498, 0.621]
actual     Down   Up
predicted           
Down         35   35
Up           76  106
QDA accuracy: 0.599 [0.539, 0.66]
actual     Down   Up
predicted           
Down         30   20
Up           81  121

The classes are balanced, with 50.8% of the training days up. The means of both lags are slightly negative on days that went up and slightly positive on days that went down: -0.040 and 0.043 for Lag1, and -0.031 and 0.034 for Lag2. The gap between the Lag1 means, 0.08, is small next to the standard deviation of Lag1 within a class, about 1.23, so the classes overlap almost completely.

ExrExercise 29 (LDA and QDA on the stock market (part 2))

Exercise 29 (LDA and QDA on the stock market (part 2)) Use the setup of Exercise 28.

How often is each model right on the 2005 days? Compare with the rule that always answers Up, and with the logistic regression of its solution.

Solution 29. LDA is right on 56% of the 252 days in 2005 (95% interval [49.8, 62.1]%), QDA on 59.9% ([53.9, 66]%), and always predicting Up on 56% ([49.8, 62.1]%). LDA does no better than the constant rule that always predicts Up, and the logistic regression of its solution, with more predictors, was right on only 48%. QDA exceeds the accuracy of the constant rule by 4 percentage points.

ExrExercise 30 (LDA and QDA on the stock market (part 3))

Exercise 30 (LDA and QDA on the stock market (part 3)) Use the setup of Exercise 28.

How much trust does the accuracy difference between the two models deserve, given that it comes from one year of data?

Solution 30. Little. Each confidence interval in Solution 29 is the accuracy plus or minus 1.96 standard errors, and the standard error is about 3.1 percentage points for QDA on 252 days, so the intervals for the three rules overlap almost entirely.

The rules are scored on the same days, so the sharper comparison is day by day. QDA exceeds the accuracy of the constant rule by 4 percentage points (95% interval [-1.5, 9.5]), an interval that includes 0. It exceeds the accuracy of LDA by 4 percentage points ([0.5, 7.4]), an interval that excludes 0.

These intervals reflect sampling variation across the test days in 2005, not across training sets, and they do not account for comparing several rules on the same days before reporting the best. Showing that QDA has higher out-of-sample accuracy would need more test days or repeated splits.

Show code
library(ggplot2)
grid <- expand.grid(
  Lag1 = seq(-4, 4, length.out = 120),
  Lag2 = seq(-4, 4, length.out = 120)
)
grid_post <- rbind(
  transform(grid, model = "LDA", p_up = predict(lda2, grid)$posterior[, "Up"]),
  transform(grid, model = "QDA", p_up = predict(qda2, grid)$posterior[, "Up"])
)
ggplot(grid_post) +
  aes(Lag1, Lag2) +
  geom_raster(aes(fill = p_up)) +
  geom_contour(aes(z = p_up), breaks = 0.5, colour = "black") +
  geom_point(
    data = da_test,
    aes(shape = Direction),
    colour = "grey20", alpha = 0.7
  ) +
  scale_fill_gradient2(midpoint = 0.5, limits = c(0, 1), name = "P(Up)") +
  facet_wrap(~model) +
  coord_cartesian(xlim = c(-4, 4), ylim = c(-4, 4))
Figure 11: Predicted probability of an up day from the LDA and QDA fits of Solution 28, over the plane of the two lagged returns. The black line and curve are the decision boundaries, where the probability is one half. The points are the days of 2005.

Two side-by-side maps of the plane of Lag1 (horizontal) and Lag2 (vertical), each from -4 to 4, shaded by the predicted probability of an up day. The shading is pale everywhere: faint blue toward the lower left, where an up day is slightly more likely than not, and faint red toward the upper right, where it is slightly less likely. In the left panel, labelled LDA, a straight black line runs from the upper left, near (-2.6, 4), to the lower right, near (3.8, -4), through the middle of the data. In the right panel, labelled QDA, the black boundary is a curve instead: it starts at the left edge near Lag2 = 3, runs down and to the right through the data, reaches its farthest right near Lag1 = 1.5 at Lag2 = -2, and turns back to the bottom edge near Lag1 = 1. About 250 points for the days of 2005, circles for down days and triangles for up days, form one round cloud centred near the origin in both panels, with both kinds of day on both sides of each boundary.

Figure 12: The graph of the predicted probability of an up day of Figure 11, as a surface over the plane of the two lagged returns. The gray plane is at probability one half, and the decision boundary is where it cuts the surface: a line for LDA and a curve for QDA.
Show code
sm_axis <- seq(-4, 4, length.out = 60)
sm_grid <- tidyr::expand_grid(Lag1 = sm_axis, Lag2 = sm_axis)
sm_n <- length(sm_axis)
sm_lda_fit <- predict(lda2, sm_grid)
sm_qda_fit <- predict(qda2, sm_grid)
sm_lda_z <- matrix(sm_lda_fit$posterior[, "Up"], nrow = sm_n)
sm_qda_z <- matrix(sm_qda_fit$posterior[, "Up"], nrow = sm_n)
sm_half <- matrix(0.5, nrow = sm_n, ncol = sm_n)
sm_stops <- c(0, 0.5, 1)
sm_cols <- c("#0072B2", "#F2F2F2", "#D55E00")
sm_scale <- list(sm_stops, sm_cols)
sm_grey <- c("#808080", "#808080")
sm_plane_stops <- c(0, 1)
sm_plane_scale <- list(sm_plane_stops, sm_grey)
sm_eye <- list(x = 1.6, y = -1.6, z = 0.9)
sm_camera <- list(eye = sm_eye)
sm_xaxis <- list(title = "Lag1")
sm_yaxis <- list(title = "Lag2")
sm_zrange <- c(0, 1)
sm_zaxis <- list(title = "P(Up)", range = sm_zrange)
sm_scene <- list(
  xaxis = sm_xaxis,
  yaxis = sm_yaxis,
  zaxis = sm_zaxis,
  camera = sm_camera
)
sm_lda_plot <- plotly::plot_ly() |>
  plotly::add_surface(
    x = sm_axis,
    y = sm_axis,
    z = sm_lda_z,
    cmin = 0,
    cmax = 1,
    colorscale = sm_scale,
    showscale = FALSE
  ) |>
  plotly::add_surface(
    x = sm_axis,
    y = sm_axis,
    z = sm_half,
    colorscale = sm_plane_scale,
    showscale = FALSE,
    opacity = 0.3,
    hoverinfo = "skip"
  ) |>
  plotly::layout(scene = sm_scene)
sm_lda_plot
Show code
import html
import numpy as np
import plotly.graph_objects as go
from sklearn.discriminant_analysis import (
    LinearDiscriminantAnalysis,
    QuadraticDiscriminantAnalysis,
)

fig_alt = r.lda_alt_sf

sm_train_3d = r.da_train
sm_x_3d = sm_train_3d[["Lag1", "Lag2"]].to_numpy()
sm_y_3d = sm_train_3d["Direction"].astype(str).to_numpy()
sm_axis_3d = np.linspace(-4, 4, 60)
sm_lag1_3d, sm_lag2_3d = np.meshgrid(sm_axis_3d, sm_axis_3d)
sm_points_3d = np.column_stack([sm_lag1_3d.ravel(), sm_lag2_3d.ravel()])
sm_scale_3d = [[0, "#0072B2"], [0.5, "#F2F2F2"], [1, "#D55E00"]]
sm_lda_model = LinearDiscriminantAnalysis().fit(sm_x_3d, sm_y_3d)
# `classes_` is sorted, so the column of "Up" comes after "Down".
sm_lda_up = sm_lda_model.predict_proba(sm_points_3d)[:, 1]
sm_lda_z_3d = sm_lda_up.reshape(sm_lag1_3d.shape)
fig = go.Figure(data=[
    go.Surface(
        x=sm_axis_3d, y=sm_axis_3d, z=sm_lda_z_3d,
        colorscale=sm_scale_3d, cmin=0, cmax=1,
        showscale=False, opacity=0.9,
    ),
    go.Surface(
        x=sm_axis_3d, y=sm_axis_3d,
        z=np.full_like(sm_lda_z_3d, 0.5),
        colorscale=[[0, "grey"], [1, "grey"]],
        showscale=False, opacity=0.3,
    ),
])
fig.update_layout(
    scene=dict(
        xaxis_title="Lag1",
        yaxis_title="Lag2",
        zaxis_title="P(Up)",
        zaxis=dict(range=[0, 1]),
    ),
);
print(
    "```{=html}\n"
    + f'<div role="group" aria-label="{html.escape(fig_alt)}">\n'
    + fig.to_html(full_html=False, include_plotlyjs="cdn")
    + "\n</div>\n```"
)
(a) The LDA surface.
Show code
sm_qda_plot <- plotly::plot_ly() |>
  plotly::add_surface(
    x = sm_axis,
    y = sm_axis,
    z = sm_qda_z,
    cmin = 0,
    cmax = 1,
    colorscale = sm_scale,
    showscale = FALSE
  ) |>
  plotly::add_surface(
    x = sm_axis,
    y = sm_axis,
    z = sm_half,
    colorscale = sm_plane_scale,
    showscale = FALSE,
    opacity = 0.3,
    hoverinfo = "skip"
  ) |>
  plotly::layout(scene = sm_scene)
sm_qda_plot
Show code
import html

fig_alt = r.qda_alt_sf

sm_qda_model = QuadraticDiscriminantAnalysis().fit(sm_x_3d, sm_y_3d)
# `classes_` is sorted, so the column of "Up" comes after "Down".
sm_qda_up = sm_qda_model.predict_proba(sm_points_3d)[:, 1]
sm_qda_z_3d = sm_qda_up.reshape(sm_lag1_3d.shape)
fig = go.Figure(data=[
    go.Surface(
        x=sm_axis_3d, y=sm_axis_3d, z=sm_qda_z_3d,
        colorscale=sm_scale_3d, cmin=0, cmax=1,
        showscale=False, opacity=0.9,
    ),
    go.Surface(
        x=sm_axis_3d, y=sm_axis_3d,
        z=np.full_like(sm_qda_z_3d, 0.5),
        colorscale=[[0, "grey"], [1, "grey"]],
        showscale=False, opacity=0.3,
    ),
])
fig.update_layout(
    scene=dict(
        xaxis_title="Lag1",
        yaxis_title="Lag2",
        zaxis_title="P(Up)",
        zaxis=dict(range=[0, 1]),
    ),
);
print(
    "```{=html}\n"
    + f'<div role="group" aria-label="{html.escape(fig_alt)}">\n'
    + fig.to_html(full_html=False, include_plotlyjs="cdn")
    + "\n</div>\n```"
)
(b) The QDA surface.
Show code
sd_axis <- seq(-4, 4, length.out = 60)
sd_grid <- tidyr::expand_grid(Lag1 = sd_axis, Lag2 = sd_axis)
sd_n <- length(sd_axis)
sd_cols_in <- c("Lag1", "Lag2")
sd_z <- list()
for (direction in c("Up", "Down")) {
  in_class <- da_train$Direction == direction
  cls <- da_train[in_class, sd_cols_in]
  cls_mean <- colMeans(cls)
  centred_df <- sweep(sd_grid, 2, cls_mean, "-")
  centred <- as.matrix(centred_df)
  sigma <- cov(cls)
  scaled <- centred %*% solve(sigma)
  quad <- rowSums(scaled * centred)
  sigma_det <- det(sigma)
  norm_const <- 2 * pi * sqrt(sigma_det)
  dens <- exp(-quad / 2) / norm_const
  weighted <- mean(in_class) * dens
  sd_z[[direction]] <- matrix(weighted, nrow = sd_n)
}
sd_up_z <- sd_z[["Up"]]
sd_down_z <- sd_z[["Down"]]
Show code
sd_stops <- c(0, 1)
sd_blue_cols <- c("#C6DBEF", "#0072B2")
sd_orange_cols <- c("#FDE0C5", "#D55E00")
sd_blue <- list(sd_stops, sd_blue_cols)
sd_orange <- list(sd_stops, sd_orange_cols)
sd_eye <- list(x = 1.6, y = -1.6, z = 0.9)
sd_xaxis <- list(title = "Lag1")
sd_yaxis <- list(title = "Lag2")
sd_zaxis <- list(title = "prior times density")
sd_camera <- list(eye = sd_eye)
sd_scene <- list(
  xaxis = sd_xaxis, yaxis = sd_yaxis, zaxis = sd_zaxis, camera = sd_camera
)
sd_legend <- list(orientation = "h", x = 0, y = -0.05)
sd_plot <- plotly::plot_ly() |>
  plotly::add_surface(
    x = sd_axis, y = sd_axis, z = sd_up_z,
    name = "Up", showlegend = TRUE, colorscale = sd_blue,
    showscale = FALSE, opacity = 0.8
  ) |>
  plotly::add_surface(
    x = sd_axis, y = sd_axis, z = sd_down_z,
    name = "Down", showlegend = TRUE, colorscale = sd_orange,
    showscale = FALSE, opacity = 0.8
  ) |>
  plotly::layout(scene = sd_scene, legend = sd_legend)
sd_plot
Show code
import numpy as np
import plotly.graph_objects as go
from scipy.stats import multivariate_normal

sd_train = r.da_train
sd_axis_py = np.linspace(-4, 4, 60)
sd_lag1, sd_lag2 = np.meshgrid(sd_axis_py, sd_axis_py)
sd_points = np.dstack([sd_lag1, sd_lag2])
fig = go.Figure()
for direction, colour in [("Up", "#0072B2"), ("Down", "#D55E00")]:
    cls = sd_train.loc[sd_train["Direction"] == direction, ["Lag1", "Lag2"]]
    prior_sd = len(cls) / len(sd_train)
    dens = multivariate_normal(cls.mean().to_numpy(), cls.cov().to_numpy())
    fig.add_trace(go.Surface(
        x=sd_axis_py, y=sd_axis_py, z=prior_sd * dens.pdf(sd_points),
        colorscale=[[0, "white"], [1, colour]], showscale=False,
        opacity=0.8, name=direction, showlegend=True));
fig.update_layout(
    scene=dict(
        xaxis_title="Lag1",
        yaxis_title="Lag2",
        zaxis_title="prior times density",
        camera=dict(eye=dict(x=1.6, y=-1.6, z=0.9)),
    ),
    legend=dict(orientation="h", x=0, y=-0.05),
);
print("```{=html}\n" + fig.to_html(full_html=False, include_plotlyjs="cdn") + "\n```")
Figure 13: The generative view of the QDA fit of Figure 12 (b): each class’s prior times its Gaussian density over the plane of the two lagged returns. By Bayes’ rule, the predicted probability of an up day is the height of the blue surface as a fraction of the sum of the two heights. The two surfaces almost coincide, so that fraction stays close to one half, as in Figure 12 (b).
ExrExercise 31 (Lab: split the Smarket data by year)

Exercise 31 (Lab: split the Smarket data by year) Split the Smarket data [5] into a training set of the days before \(2005\) and a test set of the days in \(2005\). Keep Lag1 and Lag2 as the predictors and Direction as the outcome.

TipHint

A mask on a column keeps the matching rows, and a selection of columns gives the predictors and the outcome.

Solution 31.

import pandas as pd

smarket = pd.read_csv("data/smarket.csv")
predictors = ["Lag1", "Lag2"]
is_train = smarket["Year"] < 2005
x_train = smarket.loc[is_train, predictors]
x_test = smarket.loc[~is_train, predictors]
y_train = smarket.loc[is_train, "Direction"]
y_test = smarket.loc[~is_train, "Direction"]
len(y_train), len(y_test)
(998, 252)
smarket <- readr::read_csv("data/smarket.csv", show_col_types = FALSE)
is_train <- smarket$Year < 2005
lag_cols <- c("Lag1", "Lag2")
x_train <- smarket[is_train, lag_cols]
x_test <- smarket[!is_train, lag_cols]
y_train <- smarket$Direction[is_train]
y_test <- smarket$Direction[!is_train]
n_train <- length(y_train)
n_test <- length(y_test)
c(n_train, n_test)
[1] 998 252
ExrExercise 32 (Lab: find the test error at K = 1)

Exercise 32 (Lab: find the test error at K = 1) Fit a \(K\)-nearest neighbors classifier with \(K = 1\) to the training set of Solution 31, and find its test error.

TipHint

Every estimator is fit with .fit on the training set and used with .predict on the test set. In R, class::knn() does both in one call. The test error is the fraction of test days on which the prediction differs from Direction.

Solution 32.

from sklearn.neighbors import KNeighborsClassifier

knn1_py = KNeighborsClassifier(n_neighbors=1).fit(x_train, y_train)
knn1_py_error = float((knn1_py.predict(x_test) != y_test).mean())
knn1_py_error
0.5
set.seed(1)
knn1_pred <- class::knn(x_train, x_test, y_train, k = 1)
knn1_r_error <- mean(knn1_pred != y_test)
knn1_r_error
[1] 0.5

The test error at \(K = 1\) is 0.50 in the R run and 0.50 in the Python run. Both are near \(0.5\), the error of a coin flip, which suggests that Lag1 and Lag2 carry little information about the direction of the market.

ExrExercise 33 (Lab: find the test error at K = 3)

Exercise 33 (Lab: find the test error at K = 3) Fit a \(K\)-nearest-neighbors classifier with \(K = 3\) to the training set of Solution 31, and find its test error.

TipHint

Every estimator is fit with .fit on the training set and used with .predict on the test set. In R, class::knn() does both in one call. Change only the number of neighbors from Exercise 32.

Solution 33.

knn3_py = KNeighborsClassifier(n_neighbors=3).fit(x_train, y_train)
knn3_py_error = float((knn3_py.predict(x_test) != y_test).mean())
knn3_py_error
0.46825396825396826
knn3_pred <- class::knn(x_train, x_test, y_train, k = 3)
knn3_r_error <- mean(knn3_pred != y_test)
knn3_r_error
[1] 0.4642857

The test error at \(K = 3\) is 0.46 in the R run and 0.47 in the Python run.

ExrExercise 34 (Lab: fit a naive Bayes classifier)

Exercise 34 (Lab: fit a naive Bayes classifier) Fit a Gaussian naive Bayes classifier to the training set of Solution 31, and report the estimated prior probability of each class.

TipHint

The naive Bayes assumption lets you summarize each predictor separately within each class, and the estimator interface shows where an estimator stores what it learned.

Solution 34.

from sklearn.naive_bayes import GaussianNB

nb_py = GaussianNB().fit(x_train, y_train)
dict(zip(nb_py.classes_.tolist(), nb_py.class_prior_.round(3).tolist()))
{'Down': 0.492, 'Up': 0.508}
library(discrim) # nolint: undesirable_function_linter
direction_train <- factor(y_train)
train_nb <- dplyr::bind_cols(x_train, Direction = direction_train)
nb_spec <- naive_Bayes() |>
  set_engine("naivebayes", usekernel = FALSE) |>
  set_mode("classification")
nb_r <- fit(nb_spec, Direction ~ Lag1 + Lag2, data = train_nb)
round(nb_r$fit$prior, 3)

 Down    Up 
0.492 0.508 

The priors are the fractions of days in the training set on which the market went down and up, so the two runs agree.

ExrExercise 35 (Lab: find the test error of naive Bayes)

Exercise 35 (Lab: find the test error of naive Bayes) Find the test error of the naive Bayes classifier from Solution 34 on the test set of Solution 31.

TipHint

Use .predict in Python, as in the scikit-learn estimator interface, or predict() in R, then compare the predicted classes with the actual ones on the test set.

Solution 35.

nb_py_error = float((nb_py.predict(x_test) != y_test).mean())
nb_py_error
0.40476190476190477
nb_pred <- predict(nb_r, new_data = x_test)$.pred_class
nb_r_error <- mean(nb_pred != y_test)
nb_r_error
[1] 0.4087302

The test error is 0.41 in the R run and 0.40 in the Python run.

7 Further reading

ReadingRead

James et al. [2], Chapter 4, derives these classifiers and compares them with logistic regression. Hastie et al. [8], Section 4.3, covers discriminant analysis, including its regularized versions.

Definitions on this page

References

[1]
B. Hutchinson, “DATA 471/571: Machine learning.” Western Washington University, 2024.
[2]
G. James, D. Witten, T. Hastie, and R. Tibshirani, An introduction to statistical learning with applications in R, 2nd ed. Springer, 2021. Available: https://www.statlearning.com/
[3]
Wikipedia contributors, “Bayes classifier — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Bayes_classifier
[4]
Wikipedia contributors, “Generative model — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Generative_model
[5]
G. James, D. Witten, T. Hastie, R. Tibshirani, and J. Taylor, An introduction to statistical learning: With applications in Python. Springer, 2023. doi: 10.1007/978-3-031-38747-0.
[6]
Wikipedia contributors, “Linear discriminant analysis — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Linear_discriminant_analysis
[7]
W. N. Venables and B. D. Ripley, Modern applied statistics with S, 4th ed. in Statistics and computing. Springer, 2002. doi: 10.1007/978-0-387-21706-2.
[8]
T. Hastie, R. Tibshirani, and J. Friedman, The elements of statistical learning: Data mining, inference, and prediction, 2nd ed. Springer, 2009. Available: https://hastie.su.domains/ElemStatLearn/
[9]
Wikipedia contributors, “Quadric — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Quadric
[10]
Wikipedia contributors, “Quadratic classifier — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Quadratic_classifier
[11]
Wikipedia contributors, “Conditional independence — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Conditional_independence
[12]
M. Hopkins, E. Reeber, G. Forman, and J. Suermondt, “Spambase.” UCI Machine Learning Repository, 1999. doi: 10.24432/C53G6X.
[13]
D. Barber, Bayesian reasoning and machine learning. Cambridge: Cambridge University Press, 2012. doi: 10.1017/CBO9780511804779.
[14]
Wikipedia contributors, “Naive bayes classifier — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Naive_Bayes_classifier
Back to top