Predictions and Inference for Logistic Regression

Last modified: 2026-11-19

Show code
# The Python chunks import these packages; reticulate provisions them.
library(reticulate) # nolint: undesirable_function_linter
py_packages <- c(
  "numpy", "pandas", "statsmodels", "scipy", "scikit-learn", "matplotlib",
  "plotly"
)
py_require(py_packages)

Source

This material was a section of the logistic regression page. It carries the same sources: Brian Hutchinson’s Fall 2025 lecture notes, Kameron Decker Harris’ 2022 weekly notes and Logan Sizemore’s Spring 2025 lectures, as described there.

1 Quantifying the uncertainty of the coefficients

A fitted coefficient is an estimate: another sample from the same population would give a different value, and its sampling distribution describes how much it varies. The matrix \(\qform{W}{\design}\) that Newton’s method uses to compute its step also gives the size of that variation. Where the log-likelihood curves sharply at \(\hvcoef\), \(\qform{W}{\design}\) is large and the standard errors are small; where it is nearly flat, the standard errors are large.

Remark 1 (Inference for logistic regression)

Remark 1 (Inference for logistic regression). The statistics notes’ page Inference for Logistic Regression covers the deviance, the null deviance, the likelihood ratio test of the slopes, and the Wald interval for a fitted logistic coefficient. It shows that the drop from the null deviance to the deviance is the likelihood ratio statistic. It also shows that, for a large sample, the fitted coefficients \(\hvcoef\) are approximately Gaussian with mean \(\vcoef\) and covariance \(\inv{\tp{\design} W \design}\) (its approximate distribution theorem), where \(W\) is the weight matrix in the logistic Hessian theorem. In that result the weights \(\prob_i \paren{1 - \prob_i}\) are computed from the true probabilities; the standard errors, like the Newton step, use the weights computed from the fitted probabilities instead.

2 Fitting on real data

Statistical software fits logistic regression by the Newton iteration of part 2 of the Newton and IRLS exercise. This section uses it on stock market data, to predict whether the market rises on a given day.

Dataset 1: The stock market (Smarket) data

Daily percentage returns of the S&P 500 stock index on 1250 trading days from 2001 to 2005 make up the Smarket data [1, Sec. 4.7]. The data are in data/smarket.csv, one row per trading day, copied from the ISLP Python package. The columns this course uses are:

  • Direction: the outcome, whether the index rose (Up) or fell (Down) that day.
  • Lag1 to Lag5: the percentage returns on each of the five previous trading days.
  • Volume: the number of shares traded on the previous day, in billions.
  • Year: the year of the trading day.

The Today column, the day’s own return, is not a predictor, because Direction is Down exactly when Today is negative.

The Date column, the calendar date of each trading day, is not used by the exercises; it names the day that a row describes, from 2001-01-10 to 2005-12-30. The ISLP copy has no dates. The dates were recovered by matching Today against the S&P 500 daily returns in the sp500ret data of the rugarch package [2], after converting those log returns to percentage returns. The script tools/add_smarket_dates.R does the match. It stops unless exactly one run of trading days agrees with all 1250 values of Today, to within 0.0005 percentage points, the rounding of Today.

Dataset 1: The stock market (Smarket) data
Code: read the Smarket data and show it
smarket_preview <- readr::read_csv("data/smarket.csv", show_col_types = FALSE)
smarket_preview
Code: summarize each variable of the Smarket data
dim(smarket_preview)
[1] 1250   10
Code: summarize each variable of the Smarket data
skimr::skim_without_charts(smarket_preview)

Variable type: character

skim_variable n_missing complete_rate min max empty n_unique whitespace
Direction 0 1 2 4 0 2 0

Variable type: Date

skim_variable n_missing complete_rate min max median n_unique
Date 0 1 2001-01-10 2005-12-30 2003-07-10 1250

Variable type: numeric

skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100
Year 0 1 2003.02 1.41 2001.00 2002.00 2003.00 2004.00 2005.00
Lag1 0 1 0.00 1.14 -4.92 -0.64 0.04 0.60 5.73
Lag2 0 1 0.00 1.14 -4.92 -0.64 0.04 0.60 5.73
Lag3 0 1 0.00 1.14 -4.92 -0.64 0.04 0.60 5.73
Lag4 0 1 0.00 1.14 -4.92 -0.64 0.04 0.60 5.73
Lag5 0 1 0.01 1.15 -4.92 -0.64 0.04 0.60 5.73
Volume 0 1 1.48 0.36 0.36 1.26 1.42 1.64 3.15
Today 0 1 0.00 1.14 -4.92 -0.64 0.04 0.60 5.73
Code: read the Smarket data and show it
import pandas as pd

pd.set_option("display.max_columns", None)
pd.set_option("display.width", 120)

smarket_preview = pd.read_csv("data/smarket.csv")
smarket_preview
      Year   Lag1   Lag2   Lag3   Lag4   Lag5   Volume  Today Direction        Date
0     2001  0.381 -0.192 -2.624 -1.055  5.010  1.19130  0.959        Up  2001-01-10
1     2001  0.959  0.381 -0.192 -2.624 -1.055  1.29650  1.032        Up  2001-01-11
2     2001  1.032  0.959  0.381 -0.192 -2.624  1.41120 -0.623      Down  2001-01-12
3     2001 -0.623  1.032  0.959  0.381 -0.192  1.27600  0.614        Up  2001-01-16
4     2001  0.614 -0.623  1.032  0.959  0.381  1.20570  0.213        Up  2001-01-17
...    ...    ...    ...    ...    ...    ...      ...    ...       ...         ...
1245  2005  0.422  0.252 -0.024 -0.584 -0.285  1.88850  0.043        Up  2005-12-23
1246  2005  0.043  0.422  0.252 -0.024 -0.584  1.28581 -0.955      Down  2005-12-27
1247  2005 -0.955  0.043  0.422  0.252 -0.024  1.54047  0.130        Up  2005-12-28
1248  2005  0.130 -0.955  0.043  0.422  0.252  1.42236 -0.298      Down  2005-12-29
1249  2005 -0.298  0.130 -0.955  0.043  0.422  1.38254 -0.489      Down  2005-12-30

[1250 rows x 10 columns]
Code: summarize each variable of the Smarket data
smarket_preview.info()
<class 'pandas.DataFrame'>
RangeIndex: 1250 entries, 0 to 1249
Data columns (total 10 columns):
 #   Column     Non-Null Count  Dtype  
---  ------     --------------  -----  
 0   Year       1250 non-null   int64  
 1   Lag1       1250 non-null   float64
 2   Lag2       1250 non-null   float64
 3   Lag3       1250 non-null   float64
 4   Lag4       1250 non-null   float64
 5   Lag5       1250 non-null   float64
 6   Volume     1250 non-null   float64
 7   Today      1250 non-null   float64
 8   Direction  1250 non-null   str    
 9   Date       1250 non-null   str    
dtypes: float64(7), int64(1), str(2)
memory usage: 97.8 KB
Code: summarize each variable of the Smarket data
print(smarket_preview.describe())
              Year         Lag1         Lag2         Lag3         Lag4        Lag5       Volume        Today
count  1250.000000  1250.000000  1250.000000  1250.000000  1250.000000  1250.00000  1250.000000  1250.000000
mean   2003.016000     0.003834     0.003919     0.001716     0.001636     0.00561     1.478305     0.003138
std       1.409018     1.136299     1.136280     1.138703     1.138774     1.14755     0.360357     1.136334
min    2001.000000    -4.922000    -4.922000    -4.922000    -4.922000    -4.92200     0.356070    -4.922000
25%    2002.000000    -0.639500    -0.639500    -0.640000    -0.640000    -0.64000     1.257400    -0.639500
50%    2003.000000     0.039000     0.039000     0.038500     0.038500     0.03850     1.422950     0.038500
75%    2004.000000     0.596750     0.596750     0.596750     0.596750     0.59700     1.641675     0.596750
max    2005.000000     5.733000     5.733000     5.733000     5.733000     5.73300     3.152470     5.733000
Code: summarize each variable of the Smarket data
print(smarket_preview.isna().sum())
Year         0
Lag1         0
Lag2         0
Lag3         0
Lag4         0
Lag5         0
Volume       0
Today        0
Direction    0
Date         0
dtype: int64

Exercise 1 (Predicting the direction of the stock market)

Exercise 1 (Predicting the direction of the stock market) Use the Smarket data. Fit a logistic regression for the probability that Direction is Up, with the five lagged returns and Volume as predictors, on the days before 2005 only.

Figure 1 plots the number of trading days in each year by Direction, including the days of 2005 that the fit leaves out.

Show code
sf_data <- readr::read_csv("data/smarket.csv", show_col_types = FALSE)
sf_fills <- c(Down = "#0072B2", Up = "#D55E00")
sf_data |>
  ggplot2::ggplot(ggplot2::aes(Year, fill = Direction)) +
  ggplot2::geom_bar(width = 0.6) +
  ggplot2::scale_fill_manual(values = sf_fills) +
  ggplot2::labs(x = "Year", y = "Trading days", fill = NULL)
Show code
import matplotlib.pyplot as plt
import pandas as pd

smarket_sf = pd.read_csv("data/smarket.csv")
counts_sf = smarket_sf.groupby(["Year", "Direction"]).size().unstack()
fig, ax = plt.subplots(figsize=(5, 3), layout="constrained")
ax.bar(counts_sf.index, counts_sf["Down"], color="#0072B2", width=0.6,
       label="Down");
ax.bar(counts_sf.index, counts_sf["Up"], bottom=counts_sf["Down"],
       color="#D55E00", width=0.6, label="Up");
ax.set_xlabel("Year");
ax.set_ylabel("Trading days");
ax.legend();
plt.show()
Figure 1: The number of trading days in each year of the Smarket data, split by Direction.
(a)
Stacked bar chart of the number of trading days in each year from 2001 to 2005, split into Down days in blue and Up days in vermilion. Each year from 2002 to 2005 has 252 days, and 2001 has slightly fewer, 242.
(b)
Stacked bar chart of the number of trading days in each year from 2001 to 2005, split into Down days in blue and Up days in vermilion. Each year from 2002 to 2005 has 252 days, and 2001 has slightly fewer, 242.

Solution 1.

smarket <- read.csv("data/smarket.csv")
str(smarket)
'data.frame':   1250 obs. of  10 variables:
 $ Year     : int  2001 2001 2001 2001 2001 2001 2001 2001 2001 2001 ...
 $ Lag1     : num  0.381 0.959 1.032 -0.623 0.614 ...
 $ Lag2     : num  -0.192 0.381 0.959 1.032 -0.623 ...
 $ Lag3     : num  -2.624 -0.192 0.381 0.959 1.032 ...
 $ Lag4     : num  -1.055 -2.624 -0.192 0.381 0.959 ...
 $ Lag5     : num  5.01 -1.055 -2.624 -0.192 0.381 ...
 $ Volume   : num  1.19 1.3 1.41 1.28 1.21 ...
 $ Today    : num  0.959 1.032 -0.623 0.614 0.213 ...
 $ Direction: chr  "Up" "Up" "Down" "Up" ...
 $ Date     : chr  "2001-01-10" "2001-01-11" "2001-01-12" "2001-01-16" ...
smarket <- dplyr::mutate(smarket, up = as.integer(Direction == "Up"))
train <- smarket$Year < 2005

fit <- glm(
  up ~ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume,
  data = dplyr::filter(smarket, train),
  family = binomial
)
coef(fit)
 (Intercept)         Lag1         Lag2         Lag3         Lag4         Lag5 
 0.191212621 -0.054178292 -0.045805334  0.007200118  0.006440875 -0.004222672 
      Volume 
-0.116256960 
import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

smarket_py = pd.read_csv("data/smarket.csv")
print(smarket_py.head(3))
   Year   Lag1   Lag2   Lag3   Lag4   Lag5  Volume  Today Direction        Date
0  2001  0.381 -0.192 -2.624 -1.055  5.010  1.1913  0.959        Up  2001-01-10
1  2001  0.959  0.381 -0.192 -2.624 -1.055  1.2965  1.032        Up  2001-01-11
2  2001  1.032  0.959  0.381 -0.192 -2.624  1.4112 -0.623      Down  2001-01-12
smarket_py["up"] = (smarket_py["Direction"] == "Up").astype(int)
train_py = smarket_py["Year"] < 2005

fit_py = smf.glm(
    "up ~ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume",
    data=smarket_py[train_py],
    family=sm.families.Binomial(),
).fit()
print(fit_py.params.round(3))
Intercept    0.191
Lag1        -0.054
Lag2        -0.046
Lag3         0.007
Lag4         0.006
Lag5        -0.004
Volume      -0.116
dtype: float64

Exercise 2 (Predicting the direction of the stock market (part 1b))

Exercise 2 (Predicting the direction of the stock market (part 1b)) Use the setup of Exercise 1.

Predict Direction for each day in 2005 from the fit of Exercise 1, predicting Up when the predicted probability is above \(0.5\).

Solution 2. This continues Solution 1.

test_days <- dplyr::filter(smarket, !train)
p <- predict(fit, test_days, type = "response")
pred <- ifelse(p > 0.5, "Up", "Down")
actual <- smarket$Direction[!train]

table(predicted = pred, actual = actual)
         actual
predicted Down Up
     Down   77 97
     Up     34 44
p_py = fit_py.predict(smarket_py[~train_py])
pred_py = np.where(p_py > 0.5, "Up", "Down")
actual_py = smarket_py.loc[~train_py, "Direction"]

print(pd.crosstab(pred_py, actual_py, rownames=["predicted"], colnames=["actual"]))
actual     Down  Up
predicted          
Down         77  97
Up           34  44

Exercise 3 (Predicting the direction of the stock market (part 1c))

Exercise 3 (Predicting the direction of the stock market (part 1c)) Use the setup of Exercise 1.

On what proportion of the 2005 days is the prediction of Exercise 2 correct?

Solution 3. This continues Solution 1 and Solution 2.

accuracy <- mean(pred == actual)
accuracy
[1] 0.4801587
# A 95% interval for an accuracy: the proportion plus or minus
# 1.96 standard errors, which is accurate when the number of test days
# is in the hundreds.
n_test <- length(actual)
bounds <- c(lower = -1, upper = 1)
z_crit <- qnorm(0.975)
se_accuracy <- sqrt(accuracy * (1 - accuracy) / n_test)
ci_accuracy <- accuracy + bounds * z_crit * se_accuracy
print("accuracy:", round(float(np.mean(pred_py == actual_py)), 3))
accuracy: 0.48
def wald_ci_py(p, n):
    # A 95% interval for a proportion: p plus or minus 1.96 standard errors.
    half = 1.96 * np.sqrt(p * (1 - p) / n)
    return round(float(p - half), 3), round(float(p + half), 3)

print(wald_ci_py(float(np.mean(pred_py == actual_py)), len(actual_py)))
(0.418, 0.542)

The fitted logistic regression predicts the correct label on 48% of the 252 days in 2005 (95% interval [41.8, 54.2]%).

Exercise 4 (Predicting the direction of the stock market (part 2))

Exercise 4 (Predicting the direction of the stock market (part 2)) Use the setup of Exercise 1.

Compare the proportion correct of Exercise 3 with that of the rule that predicts Up on every day.

Solution 4.

Show code
always_up <- mean(actual == "Up")
se_always_up <- sqrt(always_up * (1 - always_up) / n_test)
ci_always_up <- always_up + bounds * z_crit * se_always_up
# Both rules are scored on the same days, so the paired difference in
# correctness, day by day, has a t interval for its mean.
hit_model <- as.integer(pred == actual)
hit_up <- as.integer(actual == "Up")
diff_hit <- hit_model - hit_up
t_crit <- qt(0.975, n_test - 1)
se_diff <- sd(diff_hit) / sqrt(n_test)
ci_vs_up <- mean(diff_hit) + bounds * t_crit * se_diff
print("always Up:", round(float(np.mean(actual_py == "Up")), 3))
always Up: 0.56

Predicting Up on every day is right on 56% of the 2005 days (95% interval [49.8, 62.1]%). Each interval is the proportion plus or minus 1.96 standard errors, \(\sqrt{\eprob (1 - \eprob) / n}\). The two rules are scored on the same days, so their difference is taken day by day: the accuracy of the logistic regression minus that of the constant rule is -7.9 percentage points (95% interval [-18.2, 2.3]). The intervals describe sampling variation across the test days in 2005, not across training sets.

The fitted model is right less often than the rule that ignores every predictor, but the interval for the difference includes 0, so these 252 days do not show that the fitted model is worse. Its predicted probabilities are all between 0.44 and 0.53: no day in 2005 gets a probability of Up far from \(0.5\). Figure 2 shows that the days that went up and the days that went down get nearly the same predicted probabilities.

Show code
library(ggplot2)
ggplot(tibble::tibble(p = p, actual = actual)) +
  aes(p, actual) +
  geom_jitter(height = 0.15, width = 0, colour = "grey40") +
  geom_vline(xintercept = 0.5, linetype = "dashed") +
  labs(x = "predicted probability of Up", y = "actual direction")
Show code
import matplotlib.pyplot as plt

rng_strip = np.random.default_rng(571)
row = np.where(actual_py == "Up", 1, 0) + rng_strip.uniform(-0.15, 0.15, len(actual_py))
fig, ax = plt.subplots(figsize=(7, 3), layout="constrained")
ax.scatter(p_py, row, color="0.25", s=12);
ax.axvline(0.5, color="black", linestyle="--");
ax.set_yticks([0, 1], ["Down", "Up"]);
ax.set_xlabel("predicted probability of Up");
ax.set_ylabel("actual direction");
plt.show()
Figure 2: Predicted probability that the market goes up, for each trading day in 2005, by what the market actually did. The dashed line is the threshold of \(0.5\).

Exercise 5 (Fitting the stock market model by gradient descent)

Exercise 5 (Fitting the stock market model by gradient descent) Exercise 1 fitted the 2001 to 2004 Smarket days with R’s glm() and statsmodels’ GLM. Fit the same logistic regression with the gradient descent of the exercise on fitting by gradient descent, standardizing the six predictors with the training days’ means and standard deviations, starting from \(\vcoef = \vzero\), with step size \(\learnrate = 0.5\) for \(2{,}000\) steps.

Solution 5.

cols <- c("Lag1", "Lag2", "Lag3", "Lag4", "Lag5", "Volume")
smarket_train <- smarket |>
  dplyr::filter(train) |>
  dplyr::select(dplyr::all_of(cols))
smarket_test <- smarket |>
  dplyr::filter(!train) |>
  dplyr::select(dplyr::all_of(cols))
mu <- colMeans(smarket_train)
sdev <- apply(smarket_train, 2, sd)
# nolint start: object_name_linter.
Xs <- cbind(1, scale(smarket_train, mu, sdev))
Xte <- cbind(1, scale(smarket_test, mu, sdev))
# nolint end
y_tr <- smarket$up[train]
sig <- function(z) 0.5 * (1 + tanh(0.5 * z))

b <- rep(0, ncol(Xs))
for (i in 1:2000) {
  b <- b - 0.5 * drop(t(Xs) %*% (sig(drop(Xs %*% b)) - y_tr)) / length(y_tr)
}
b
                     Lag1         Lag2         Lag3         Lag4         Lag5 
 0.032104744 -0.066621178 -0.056325934  0.008874033  0.007939830 -0.005248037 
      Volume 
-0.031112330 
cols = ["Lag1", "Lag2", "Lag3", "Lag4", "Lag5", "Volume"]
tr = smarket_py[train_py]
mu, sd = tr[cols].mean(), tr[cols].std()
Xs_py = np.column_stack([np.ones(len(tr)), ((tr[cols] - mu) / sd).to_numpy()])
Xte_py = np.column_stack(
    [np.ones((~train_py).sum()), ((smarket_py.loc[~train_py, cols] - mu) / sd).to_numpy()]
)
y_tr_py = tr["up"].to_numpy()
sig = lambda z: 0.5 * (1 + np.tanh(0.5 * z))

b_py = np.zeros(Xs_py.shape[1])
for _ in range(2000):
    b_py = b_py - 0.5 * Xs_py.T @ (sig(Xs_py @ b_py) - y_tr_py) / len(y_tr_py)
print("standardized:", np.round(b_py, 3))
standardized: [ 0.032 -0.067 -0.056  0.009  0.008 -0.005 -0.031]

Figure 3 plots the average log loss at each step of the loop above.

Show code
gdl_steps <- 2000
gdl_loss <- numeric(gdl_steps)
gdl_ncol <- ncol(Xs)
gdl_b <- rep(0, gdl_ncol)
for (i in seq_len(gdl_steps)) {
  gdl_z <- drop(Xs %*% gdl_b)
  gdl_p <- sig(gdl_z)
  gdl_pt <- ifelse(y_tr == 1, gdl_p, 1 - gdl_p)
  gdl_loss[i] <- mean(-log(gdl_pt))
  gdl_grad <- drop(t(Xs) %*% (gdl_p - y_tr)) / length(y_tr)
  gdl_b <- gdl_b - 0.5 * gdl_grad
}
gdl_seq <- seq_len(gdl_steps)
gdl_path <- tibble::tibble(step = gdl_seq, loss = gdl_loss)
gdl_path |>
  ggplot2::ggplot(ggplot2::aes(step, loss)) +
  ggplot2::geom_line(colour = "#0072B2") +
  ggplot2::scale_x_log10() +
  ggplot2::labs(x = "Step (log scale)", y = "Average log loss")
Show code
import matplotlib.pyplot as plt
import numpy as np

b_gdl = np.zeros(Xs_py.shape[1])
loss_gdl = []
for _ in range(2000):
    p_gdl = sig(Xs_py @ b_gdl)
    p_true = np.where(y_tr_py == 1, p_gdl, 1 - p_gdl)
    loss_gdl.append(np.mean(-np.log(p_true)))
    grad_gdl = Xs_py.T @ (p_gdl - y_tr_py) / len(y_tr_py)
    b_gdl = b_gdl - 0.5 * grad_gdl

fig, ax = plt.subplots(figsize=(5, 3), layout="constrained")
ax.plot(np.arange(1, 2001), loss_gdl, color="#0072B2");
ax.set_xscale("log");
ax.set_xlabel("Step (log scale)");
ax.set_ylabel("Average log loss");
plt.show()
Figure 3: The average log loss of the training days at each step of gradient descent.
(a)
Line plot of the average log loss on the training days against the gradient descent step, from 1 to 2000 on a logarithmic horizontal axis. The curve starts at about 0.6931, which is log 2, falls over the first 20 steps to about 0.6919, and is flat from there to step 2000.
(b)
Line plot of the average log loss on the training days against the gradient descent step, from 1 to 2000 on a logarithmic horizontal axis. The curve starts at about 0.6931, which is log 2, falls over the first 20 steps to about 0.6919, and is flat from there to step 2000.

Exercise 6 (Fitting the stock market model by gradient descent (part 1b))

Exercise 6 (Fitting the stock market model by gradient descent (part 1b)) Use the setup of Exercise 5.

Convert the coefficients of the fit of Exercise 5 back to the original units of the predictors.

Solution 6. This continues Solution 5.

slopes <- b[-1] / sdev
coef_gd <- c(b[1] - sum(slopes * mu), slopes)
coef_lib <- coef(fit)
rbind( # nolint: undesirable_function_linter.
  gradient_descent = coef_gd, library_fit = coef_lib
)
                                  Lag1        Lag2        Lag3        Lag4
gradient_descent 0.1912126 -0.05417829 -0.04580533 0.007200118 0.006440875
library_fit      0.1912126 -0.05417829 -0.04580533 0.007200118 0.006440875
                         Lag5    Volume
gradient_descent -0.004222672 -0.116257
library_fit      -0.004222672 -0.116257
log_odds <- drop(Xte %*% b)
same_class <- (sig(log_odds) > 0.5) == (p > 0.5)
same <- mean(same_class)
same
[1] 1
slopes_py = b_py[1:] / sd.to_numpy()
intercept_py = b_py[0] - np.sum(slopes_py * mu.to_numpy())
print("gradient descent:", np.round(np.r_[intercept_py, slopes_py], 3))
gradient descent: [ 0.191 -0.054 -0.046  0.007  0.006 -0.004 -0.116]
print("library fit:     ", fit_py.params.round(3).to_numpy())
library fit:      [ 0.191 -0.054 -0.046  0.007  0.006 -0.004 -0.116]
same_py = (sig(Xte_py @ b_py) > 0.5) == (p_py.to_numpy() > 0.5)
print("same predictions:", same_py.mean())
same predictions: 1.0

Exercise 7 (Fitting the stock market model by gradient descent: how close?)

Exercise 7 (Fitting the stock market model by gradient descent: how close?) Use the setup of Exercise 5.

What is the largest absolute difference between the gradient descent coefficients and those of Exercise 1?

Solution 7. The largest absolute difference between the two sets of coefficients is 4.3e-10.

Exercise 8 (Fitting the stock market model by gradient descent (part 2))

Exercise 8 (Fitting the stock market model by gradient descent (part 2)) Use the setup of Exercise 5.

On what proportion of the 2005 days do the two fits make the same prediction?

Solution 8. The two fits make the same prediction on 100% of the 2005 days. Gradient descent ends near the glm() fit for three reasons.

Show code
# The curvature bound L of the average log loss for the standardized design.
gd_lip <- max(eigen(crossprod(Xs) / nrow(Xs), only.values = TRUE)$values) / 4
c(L = gd_lip, step_bound = 2 / gd_lip)
         L step_bound 
 0.2823152  7.0842810 

Standardizing the predictors puts the coefficients on similar scales, so one step size suits all of them.

Watch

Hastie and Tibshirani fit logistic regression in code in their Python lab [3].

Read

The Chapter 4 lab of James et al. [1], Ch04-classification-lab.ipynb in ISLP_labs, fits the same logistic regression to the Smarket data, and then compares it with four other classifiers on the same data [1, Sec. 4.7]. The discriminant analysis chapter names three of them. The classification lab works the logistic regression part.

3 When the coefficients diverge

The fits so far assumed that the likelihood has a finite maximizer. When a hyperplane puts every observation of one class on one side and every observation of the other class on the other side, the likelihood has no finite maximizer [4]. Any \(\ninputvars + 1\) points in general position in \(\reals^{\ninputvars}\) can be split this way, whatever their classes, so such a split is common when the sample is small relative to the number of predictors. Fitting software still stops and reports finite coefficients, and the Wald test and the likelihood ratio test reach opposite conclusions.

Exercise 9 (When the data separate perfectly)

Exercise 9 (When the data separate perfectly) A second trial, of six patients, records a dose \(\predvar\) and recovery \(y\):

Table 1: The dose and recovery of each of six patients.
\(\predvar\) \(-3\) \(-2\) \(-1\) \(+1\) \(+2\) \(+3\)
\(y\) \(0\) \(0\) \(0\) \(1\) \(1\) \(1\)

Every patient with a negative dose failed to recover, and every patient with a positive dose recovered.

Show, as in part 3 of the worked fit exercise, that the score for \(\coef{0}\) is \(0\) at \(\coef{0} = 0\) for every \(\coef{1}\).

Figure 4 plots the six patients.

Show code
sep_dose <- c(-3, -2, -1, 1, 2, 3)
sep_y <- c(0, 0, 0, 1, 1, 1)
sep_recovery <- factor(sep_y)
sep_fig <- tibble::tibble(dose = sep_dose, y = sep_y, recovery = sep_recovery)
sep_colours <- c("0" = "#0072B2", "1" = "#D55E00")
sep_shapes <- c("0" = 16, "1" = 17)
sep_breaks <- c(0, 1)
sep_fig |>
  ggplot2::ggplot(ggplot2::aes(dose, y, colour = recovery, shape = recovery)) +
  ggplot2::geom_point(size = 3) +
  ggplot2::scale_colour_manual(values = sep_colours, guide = "none") +
  ggplot2::scale_shape_manual(values = sep_shapes, guide = "none") +
  ggplot2::scale_y_continuous(breaks = sep_breaks) +
  ggplot2::labs(x = "Dose", y = "Recovery")
Show code
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(5, 2.5), layout="constrained")
dose_sep = [-3, -2, -1]
ax.plot(dose_sep, [0, 0, 0], "o", color="#0072B2", markersize=8);
ax.plot([1, 2, 3], [1, 1, 1], "^", color="#D55E00", markersize=8);
ax.set_yticks([0, 1]);
ax.set_xlabel("Dose");
ax.set_ylabel("Recovery");
plt.show()
Figure 4: The six patients’ doses and recoveries. Every patient with a negative dose has recovery \(0\), and every patient with a positive dose has recovery \(1\).
(a)
Strip plot of six patients. The horizontal axis is the dose, from -3 to 3. Three blue circles at doses -3, -2 and -1 sit at recovery 0, and three vermilion triangles at doses 1, 2 and 3 sit at recovery 1.
(b)
Strip plot of six patients. The horizontal axis is the dose, from -3 to 3. Three blue circles at doses -3, -2 and -1 sit at recovery 0, and three vermilion triangles at doses 1, 2 and 3 sit at recovery 1.

Solution 9. In Exercise 9, the doses of the three patients with \(\outvar = 0\) are the negatives of the doses of the three with \(\outvar = 1\). At \(\coef{0} = 0\), \(\prob_i = \sigmoidf{\coef{1} \predvar_i}\), and \(\sigmoidf{-z} + \sigmoidf{z} = 1\) (part 2 of the sigmoid properties exercise), so

\[ \begin{aligned} \sumin \prob_i &= \sum_{k=1}^{3} \sb{\sigmoidf{-k\coef{1}} + \sigmoidf{k\coef{1}}} && \text{(pair dose } -k \text{ with dose } k \text{)} \\ &= 3 && \text{(each pair sums to } 1 \text{)} \\ &= \sumin \outvar_i. \end{aligned} \]

So the score for \(\coef{0}\), \(\sumin \paren{\outvar_i - \prob_i}\), is \(0\) for every \(\coef{1}\).


Exercise 10 (When the data separate perfectly: the log-likelihood)

Exercise 10 (When the data separate perfectly: the log-likelihood) Use the setup of Exercise 9.

Show that with \(\coef{0} = 0\) the log-likelihood is

\[ \llik(\coef{1}) = 2\log\sigmoidf{\coef{1}} + 2\log\sigmoidf{2\coef{1}} + 2\log\sigmoidf{3\coef{1}}. \]

Solution 10. With \(\coef{0} = 0\), a patient with \(\outvar_i = 1\) and dose \(k\) contributes \(\log\sigmoidf{k\coef{1}}\). A patient with \(\outvar_i = 0\) and dose \(-k\) contributes

\[ \begin{aligned} \log\sb{1 - \sigmoidf{-k\coef{1}}} &= \log\sigmoidf{k\coef{1}}. && \text{(} 1 - \sigmoidf{-z} = \sigmoidf{z} \text{)} \end{aligned} \]

Each \(k \in \set{1, 2, 3}\) occurs once in each group, so each \(\log\sigmoidf{k\coef{1}}\) appears twice, which gives the stated \(\llik\).

Exercise 11 (When the data separate perfectly: an increasing log-likelihood)

Exercise 11 (When the data separate perfectly: an increasing log-likelihood) Use the setup of Exercise 10.

Show that \(\llik(\coef{1})\) is strictly increasing.

Solution 11. For each \(k\), the derivative of \(\log\sigmoidf{kz}\) with respect to \(z\) is \(k\sb{1 - \sigmoidf{kz}}\), so

\[ \begin{aligned} \score(\coef{1}) &= 2\sb{1 - \sigmoidf{\coef{1}}} + 4\sb{1 - \sigmoidf{2\coef{1}}} + 6\sb{1 - \sigmoidf{3\coef{1}}} && \text{(differentiate each term)} \\ &= 2\sigmoidf{-\coef{1}} + 4\sigmoidf{-2\coef{1}} + 6\sigmoidf{-3\coef{1}} && \text{(} 1 - \sigmoidf{z} = \sigmoidf{-z} \text{)} \\ &> 0 && \text{(} \sigmoid > 0 \text{)} \end{aligned} \]

for every \(\coef{1}\), so \(\llik\) is strictly increasing.

Exercise 12 (When the data separate perfectly: the supremum)

Exercise 12 (When the data separate perfectly: the supremum) Use the setup of Exercise 11.

Find \(\sup_{\coef{1}} \llik(\coef{1})\).

Solution 12. As \(\coef{1} \to \infty\), each \(\sigmoidf{k\coef{1}} \to 1\), so each \(\log\sigmoidf{k\coef{1}} \to 0\). So \(\sup_{\coef{1}} \llik(\coef{1}) = 0\), and the supremum of the likelihood is \(1\).

Exercise 13 (When the data separate perfectly: is the supremum attained?)

Exercise 13 (When the data separate perfectly: is the supremum attained?) Use the setup of Exercise 11.

Is the supremum of the log-likelihood attained at any \((\coef{0}, \coef{1})\)?

Solution 13. No. At any finite \((\coef{0}, \coef{1})\), every fitted probability is strictly between \(0\) and \(1\), so each of the six terms of the log-likelihood is negative, and \(\llik(\coef{0}, \coef{1}) < 0\). So no \((\coef{0}, \coef{1})\) maximizes the likelihood, and the maximum likelihood estimate does not exist.


Definition 1 (Separated data (completely separated data))

Definition 1 (Separated data (completely separated data)) A dataset \((\vx_1, \outvar_1), \ldots, (\vx_\nobs, \outvar_\nobs)\) with \(\outvar_i \in \set{0, 1}\), where each \(\vx_i\) includes the leading \(1\) for the intercept, is separated, also called completely separated [4], when some \(\vcoef\) has \(\dprod{\vcoef}{\vx_i} > 0\) for every \(i\) with \(\outvar_i = 1\) and \(\dprod{\vcoef}{\vx_i} < 0\) for every \(i\) with \(\outvar_i = 0\). Such a \(\vcoef\) is said to separate the data.

Example 1 (A separated dataset and one that is not)

Example 1 (A separated dataset and one that is not) The six patients of Exercise 9 are separated: \(\vcoef = \tp{(0, 1)}\) gives \(\dprod{\vcoef}{\vx_i} = \predvar_i\), which is \(1\), \(2\) or \(3\) for the patients who recovered and \(-1\), \(-2\) or \(-3\) for those who did not.

The six patients of the worked fit exercise are not separated. Each group contains a patient who recovered and a patient who did not, and those two patients have the same \(\vx_i = \tp{(1, \predvar_i)}\), so \(\dprod{\vcoef}{\vx_i}\) cannot be positive for one and negative for the other.

Definition 2 (Quasi-completely separated data)

Definition 2 (Quasi-completely separated data) A dataset with \(\outvar_i \in \set{0, 1}\) is quasi-completely separated [4] when it is not separated, but some \(\vcoef \ne \vzero\) has \(\dprod{\vcoef}{\vx_i} \ge 0\) for every \(i\) with \(\outvar_i = 1\) and \(\dprod{\vcoef}{\vx_i} \le 0\) for every \(i\) with \(\outvar_i = 0\).

Example 2 (Four patients, quasi-completely separated)

Example 2 (Four patients, quasi-completely separated) Take four patients with doses \(\predvar = -1, 0, 0, 1\) and recoveries \(\outvar = 0, 0, 1, 1\). They are not separated: the two patients with dose \(0\) have the same \(\vx_i\) and different outcomes. The vector \(\vcoef = \tp{(0, 1)}\) gives \(\dprod{\vcoef}{\vx_i} = -1, 0, 0, 1\), which is \(\ge 0\) for both patients who recovered and \(\le 0\) for both who did not, so the data are quasi-completely separated.

quasi <- data.frame(x = c(-1, 0, 0, 1), y = c(0, 0, 1, 1))
quasi_fit <- glm(y ~ x, family = binomial, data = quasi)
summary(quasi_fit)$coefficients
                Estimate  Std. Error      z value Pr(>|z|)
(Intercept) 8.146903e-18    1.414214 5.760730e-18 1.000000
x           1.956607e+01 7604.235498 2.573049e-03 0.997947

glm() reports a slope of 19.6, with a standard error of 7604: the same symptoms as for the separated trial of Exercise 9. For quasi-completely separated data, too, the maximum likelihood estimate does not exist [4].

Exercise 14 (When the data separate perfectly (part 2))

Exercise 14 (When the data separate perfectly (part 2)) Use the setup of Exercise 9.

Now take any separated dataset, and suppose \(\vcoef\) separates it. Show that replacing \(\vcoef\) by \(c\vcoef\), for any \(c > 1\), strictly increases the likelihood.

Solution 14. By the margin form theorem, each observation’s contribution to the log-likelihood is \(-\log\paren{1 + \e{-m_i}}\), with the margin \(m_i = \signoutvar_i \dprod{\vcoef}{\vx_i}\) and \(\signoutvar_i = 2\outvar_i - 1 \in \set{-1, +1}\), so

\[ \llik(\vcoef) = -\sumin \log\paren{1 + \e{-m_i}}. \]

This margin of an observation is not the margin of a separating hyperplane, which does not change when \(\vcoef\) is scaled.

Separation says that every margin is positive: \(m_i > 0\) for all \(i\). Scaling \(\vcoef\) to \(c\vcoef\) multiplies every \(m_i\) by \(c\), so for \(c > 1\) every \(m_i\) increases. Each term \(-\log\paren{1 + \e{-m}}\) is strictly increasing in \(m\), so every term increases. Hence \(\llik(c\vcoef) > \llik(\vcoef)\).

Exercise 15 (When the data separate perfectly: the supremum in general)

Exercise 15 (When the data separate perfectly: the supremum in general) Use the setup of Exercise 14.

Find the supremum of the log-likelihood.

Solution 15. Every term \(-\log\paren{1 + \e{-m_i}}\) of Solution 14 is negative, so \(\llik < 0\) at every finite parameter vector. Along the ray \(c\vcoef\), each term tends to \(0\) as \(c \to \infty\), because \(c m_i \to \infty\). So the supremum of the log-likelihood is \(0\), and the supremum of the likelihood is \(1\).

Exercise 16 (When the data separate perfectly: no maximizer in general)

Exercise 16 (When the data separate perfectly: no maximizer in general) Use the setup of Exercise 15.

Show that no finite parameter vector attains that supremum.

Solution 16. In Solution 15, every term of \(\llik\) is negative, so \(\llik < 0\) at every finite parameter vector, while the supremum is \(0\). So no finite parameter vector attains the supremum of the log-likelihood, or that of the likelihood. Solution 12 and Solution 13 were this argument for one six-patient dataset.

Exercise 17 (When the data separate perfectly (part 3))

Exercise 17 (When the data separate perfectly (part 3)) Use the setup of Exercise 9.

At \(\coef{0} = 0\), the observed information \(\tp{\design} W \design\) is diagonal, and its entry for \(\coef{1}\) is

\[ \oinff{\coef{1}} = \sumin \predvar_i^2\,\prob_i(1 - \prob_i). \]

Evaluate \(\oinff{\coef{1}}\) at \(\coef{1} = 1, 2, 5, 10, 20\), and the standard error \(1/\sqrt{\oinff{\coef{1}}}\) that a fitting routine would report there.

Solution 17. In Exercise 17, the off-diagonal entry of \(\tp{\design} W \design\) is \(\sumin \predvar_i \prob_i (1 - \prob_i)\), which is \(0\) at \(\coef{0} = 0\) because the doses \(k\) and \(-k\) have the same weight \(\sigmoidf{k\coef{1}}\sigmoidf{-k\coef{1}}\). So the standard error of \(\coef{1}\), the square root of the corresponding diagonal entry of \(\inv{\tp{\design} W \design}\), is \(1/\sqrt{\oinff{\coef{1}}}\). At the five values:

Show code
sep <- tibble::tibble(x = c(-3, -2, -1, 1, 2, 3), y = c(0, 0, 0, 1, 1, 1))
sep_info <- function(b1) {
  p <- plogis(b1 * sep$x)
  sum(sep$x^2 * p * (1 - p))
}
sep_b1 <- c(1, 2, 5, 10, 20)
sep_i <- vapply(sep_b1, sep_info, numeric(1))
sep_se <- 1 / sqrt(sep_i)

# Typeset a number for the page: plain if it is not tiny,
# and as $m \times 10^{e}$ when its exponent is -4 or lower.
sep_num <- function(v, digits = 3) {
  v <- signif(v, digits)
  e <- floor(log10(abs(v)))
  if (e > -4) {
    return(format(v))
  }
  sprintf("$%s \\times 10^{%d}$", format(signif(v / 10^e, digits)), e)
}
sep_nums <- function(v, digits = 3) {
  vapply(v, sep_num, "", digits = digits)
}

knitr::kable(
  tibble::tibble(
    b1 = sep_b1,
    info = sep_nums(sep_i),
    se = sep_nums(sep_se)
  ),
  col.names = c(
    "$\\coef{1}$", "$\\oinf(\\coef{1})$", "$\\mathrm{SE} = 1/\\sqrt{\\oinf}$"
  ),
  align = "r"
)
\(\coef{1}\) \(\oinf(\coef{1})\) \(\mathrm{SE} = 1/\sqrt{\oinf}\)
1 2.05 0.699
2 0.396 1.59
5 0.0137 8.55
10 \(9.08 \times 10^{-5}\) 105
20 \(4.12 \times 10^{-9}\) 15600

The information tends to \(0\) as \(\coef{1}\) grows, because every \(\prob_i\) tends to \(0\) or \(1\), where \(\prob_i(1-\prob_i)\) is \(0\), so the standard error grows without bound. The slope of \(\llik\), the score of Solution 11, also tends to \(0\): far out along \(\coef{1}\) the log-likelihood is nearly flat.

Example 3 (When the data separate perfectly: what software reports)

Example 3 (When the data separate perfectly: what software reports) Fitting the six patients of Exercise 9 with R’s glm():

sep_fit <- glm(y ~ x, family = binomial, data = sep)
sep_b1_hat <- coef(sep_fit)[["x"]]
sep_p_hat <- summary(sep_fit)$coefficients["x", "Pr(>|z|)"]
# The slope's p-value, formatted by insight::format_p()
sep_p_printed <- insight::format_p(sep_p_hat)
parameters::parameters(sep_fit) |> parameters::print_md(digits = 3)
Parameter Log-Odds SE 95% CI z p
(Intercept) -9.013e-08 48419.140 (-94899.770, 94899.770) -1.862e-12 > .999
x 23.268 48418.863 (-94875.958, 94922.495) 4.806e-04 > .999

R warns, “fitted probabilities numerically 0 or 1 occurred”. That is, some fitted probabilities are within rounding error of \(0\) or \(1\).

Example 4 (When the data separate perfectly: why the fit stops)

Example 4 (When the data separate perfectly: why the fit stops) In Example 3, glm() stops and reports a slope. Far out along \(\coef{1}\) the log-likelihood is nearly flat (Solution 17). There the gradient and the change in fit are both small, so a fitting routine stops there, at a \(\coef{1}\) set by its tolerance, without having found a maximum. glm() stops after 24 iterations, when the relative change in the deviance falls below its tolerance, epsilon = 1e-8 by default. The reported slope \(23.27\) is set by that tolerance; it is not an estimate of a finite parameter. Running the same fit with a tighter tolerance:

sep_tight <- suppressWarnings(
  glm(y ~ x, family = binomial, data = sep,
      control = glm.control(epsilon = 1e-14, maxit = 100))
)
c(slope = coef(sep_tight)[["x"]], iterations = sep_tight$iter)
     slope iterations 
  30.68281   32.00000 

returns a larger slope, \(30.68\), after 32 iterations.

Exercise 18 (When the data separate perfectly (part 4))

Exercise 18 (When the data separate perfectly (part 4)) Use the setup of Exercise 17.

The Wald test of \(H_0: \coef{1} = 0\) uses \(z = \coef{1}\sqrt{\oinff{\coef{1}}}\), the slope divided by its standard error, evaluated at the fitted slope. Evaluate \(z\) and its two-sided \(p\)-value at the five values of Exercise 17.

Solution 18. In Exercise 18, the Wald statistic \(z = \coef{1}\sqrt{\oinff{\coef{1}}}\) first rises and then falls as \(\coef{1}\) grows:

Show code
sep_z <- sep_b1 * sqrt(sep_i)
sep_p <- 2 * pnorm(-sep_z)
sep_wald <- rbind(
  "$z$" = sep_nums(sep_z),
  "$p$" = sep_nums(sep_p)
)
colnames(sep_wald) <- paste0("$\\coef{1} = ", sep_b1, "$")
sep_peak <- optimize(
  \(b1) b1 * sqrt(sep_info(b1)),
  interval = c(0, 10),
  maximum = TRUE
)
sep_g2 <- 12 * log(2)
sep_g2_p <- pchisq(sep_g2, df = 1, lower.tail = FALSE)
knitr::kable(sep_wald, align = "r")
\(\coef{1} = 1\) \(\coef{1} = 2\) \(\coef{1} = 5\) \(\coef{1} = 10\) \(\coef{1} = 20\)
\(z\) 1.43 1.26 0.584 0.0953 0.00128
\(p\) 0.153 0.208 0.559 0.924 0.999

Its largest value is about \(1.44\), at \(\coef{1} \approx 1.12\), which is below \(1.96\), the two-sided 5% critical value, and after that it decreases toward \(0\). So beyond \(\coef{1} \approx 1.12\), the further the fit moves toward the supremum of the likelihood, the larger the Wald \(p\)-value. R’s fit in Example 3 gives the slope a \(p\)-value of \(0.9996\), printed as \(p > .999\), on data where the sign of the dose predicts the outcome without error.

Exercise 19 (When the data separate perfectly: the likelihood ratio test)

Exercise 19 (When the data separate perfectly: the likelihood ratio test) Use the setup of Exercise 18.

Compute the likelihood ratio statistic \(G^2 = 2\sb{\sup\llik - \llik(0, 0)}\) for \(H_0: \coef{1} = 0\), and its \(p\)-value from the \(\chi^2_1\) distribution.

Solution 19. Under \(H_0\) the slope is \(0\), and the maximum likelihood estimate of the intercept is \(\logitf{3/6} = 0\), so the null fit has log-likelihood

\[ \begin{aligned} \llik(0, 0) &= 6\log\tfrac12 \\ &= -6\log 2. \end{aligned} \]

The unrestricted fit has no maximizer, so the supremum \(\sup\llik = 0\) takes the place of the maximized log-likelihood in the likelihood ratio statistic:

\[ \begin{aligned} G^2 &= 2\sb{0 - (-6\log 2)} \\ &= 12\log 2 \\ &= 8.3178. \end{aligned} \]

Against the \(\chi^2_1\) distribution, \(G^2\) gives \(p = 0.0039\). \(G^2\) is the null deviance minus the deviance of the fit, and R’s fit reports a null deviance of \(8.3178\) and a deviance of \(3.1 \times 10^{-10}\).

Exercise 20 (When the data separate perfectly: which test to use)

Exercise 20 (When the data separate perfectly: which test to use) Use the setup of Exercise 19.

The Wald test of Exercise 18 and the likelihood ratio test of Exercise 19 reach opposite conclusions. Which of the two should you use here?

Solution 20. Use the likelihood ratio test. The Wald statistic is the slope divided by its standard error, and both grow without bound as \(\coef{1} \to \infty\). The standard error \(1/\sqrt{\oinff{\coef{1}}}\) grows exponentially, faster than the slope, so their ratio tends to \(0\). The Wald test uses the curvature of \(\llik\) at the fitted slope, far out, where \(\llik\) is nearly flat. The likelihood ratio test uses the change in log-likelihood between \(\coef{1} = 0\) and the fit, which measures how much better the separated fit describes the data.

On a separated or nearly separated dataset, then, a large reported standard error is a symptom of separation, not a sign of weak evidence.

Exercise 21 (When the data separate perfectly: how far to trust the p-value)

Exercise 21 (When the data separate perfectly: how far to trust the p-value) Use the setup of Exercise 19.

The \(p\)-value of Solution 19 relies on the \(\chi^2_1\) approximation. With only six patients, compute an exact \(p\)-value instead, by conditioning on the number of recoveries.

Solution 21. The trial of Exercise 9 has \(3\) recoveries. Under \(H_0\), each of the \(\binom{6}{3} = 20\) ways to choose which three patients recover is equally likely. Only the observed choice gives the largest possible \(\sumin \predvar_i \outvar_i\), so the one-sided exact \(p\)-value is \(1/20 = 0.05\). The two-sided one is 0.1; it also counts the mirror-image outcome. Both are weaker evidence than the \(\chi^2_1\) value of Solution 19 suggests, but they point the same way.


Definition 3 (Hauck–Donner effect)

Definition 3 (Hauck–Donner effect) In logistic regression, the Hauck–Donner effect is the decrease of the Wald statistic \(z = \hcoef{j} / \HSE{\hcoef{j}}\) for \(H_0: \coef{j} = 0\) toward \(0\) as \(\abs{\hcoef{j}}\) grows large. It happens because the standard error grows faster than \(\abs{\hcoef{j}}\) [5]. As a result, the Wald test can fail to reject \(H_0\) when the coefficient is far from \(0\).

Example 5 (The Hauck–Donner effect in the separated trial)

Example 5 (The Hauck–Donner effect in the separated trial) Take the six separated patients of Exercise 9, with \(\coef{0} = 0\). The Wald statistic for the slope is \(\coef{1}\sqrt{\oinff{\coef{1}}}\) (Exercise 18).

Show code
# The Wald z for the slope of the six separated patients, at beta_0 = 0.
hd_z <- function(b1) {
  x <- c(-3, -2, -1, 1, 2, 3)
  p <- plogis(b1 * x)
  b1 * sqrt(sum(x^2 * p * (1 - p)))
}
hd_z_values <- c(slope_2 = hd_z(2), slope_20 = hd_z(20))
hd_p_values <- 2 * pnorm(-hd_z_values)
rbind(z = hd_z_values, p = hd_p_values)
    slope_2    slope_20
z 1.2580715 0.001284104
p 0.2083659 0.998975434

A fitting routine that stopped at the estimate \(\hcoef{1} = 2\) would report a Wald statistic of \(1.26\), with \(p = 0.208\). One that stopped at \(\hcoef{1} = 20\) would report \(0.0013\), with \(p = 0.999\). The estimate ten times further from \(0\) gets the smaller \(z\) and the larger \(p\)-value.

Exercise 22 (When the data separate perfectly (part 5))

Exercise 22 (When the data separate perfectly (part 5)) Use the setup of Exercise 10.

Consider the ridge-penalized log-likelihood \(f(\coef{1}) = \llik(\coef{1}) - \tfrac{\regpar}{2}\coef{1}^2\), with \(\coef{0} = 0\) as before. Show that \(f\) has a unique maximizer for every \(\regpar > 0\).

Solution 22. Every \(\log\sigmoid\) is negative, so \(\llik < 0\) everywhere, and therefore \(f(\coef{1}) < -\tfrac{\regpar}{2}\coef{1}^2\), which tends to \(-\infty\) as \(\abs{\coef{1}} \to \infty\). A continuous function that tends to \(-\infty\) in both directions attains a maximum, so a maximizer exists. It is unique because \(f\) is strictly concave: \(f''(\coef{1}) = -\oinff{\coef{1}} - \regpar\), and \(\oinff{\coef{1}} > 0\) (Exercise 17).

Exercise 23 (When the data separate perfectly: the penalized slope)

Exercise 23 (When the data separate perfectly: the penalized slope) Use the setup of Exercise 22.

Find the maximizer \(\hcoef{1}\) of \(f\) for \(\regpar = 1\), \(0.1\) and \(0.01\).

Solution 23. Setting \(f'(\coef{1}) = 0\), with the score of Solution 11, gives

\[ 2\sigmoidf{-\coef{1}} + 4\sigmoidf{-2\coef{1}} + 6\sigmoidf{-3\coef{1}} = \regpar\coef{1}. \]

The left-hand side is positive, so the root is positive. For \(\coef{1} \ge 0\) the left-hand side decreases from \(6\) at \(\coef{1} = 0\) toward \(0\), while the right-hand side increases from \(0\), so the two sides are equal at exactly one point. Solving numerically:

Show code
sep_lambda <- c(1, 0.1, 0.01)
sep_ridge_fit <- function(lambda) {
  uniroot(
    \(b1) 2 * plogis(-b1) + 4 * plogis(-2 * b1) + 6 * plogis(-3 * b1) - lambda * b1,
    interval = c(0, 100),
    tol = 1e-10
  )$root
}
sep_ridge <- vapply(sep_lambda, sep_ridge_fit, numeric(1))
sep_ridge_tbl <- rbind("$\\hcoef{1}$" = round(sep_ridge, 3))
colnames(sep_ridge_tbl) <- paste0("$\\lambda = ", sep_lambda, "$")
knitr::kable(sep_ridge_tbl, align = "r")
\(\lambda = 1\) \(\lambda = 0.1\) \(\lambda = 0.01\)
\(\hcoef{1}\) 1.104 2.295 3.946

Exercise 24 (When the data separate perfectly: what the penalty costs)

Exercise 24 (When the data separate perfectly: what the penalty costs) Use the setup of Exercise 23.

What does the penalty achieve, and at what cost?

Solution 24. In Solution 22, the ridge penalty gives a finite, unique maximizer.

The cost is that the maximizer depends strongly on \(\regpar\). It is 3.6 times larger at \(\regpar = 0.01\) than at \(\regpar = 1\), and it tends to infinity as \(\regpar \to 0\), because the likelihood alone puts no upper bound on the slope. The penalty adds no information about the slope from the data; it adds a preference for smaller coefficients, and its maximizer reflects that preference as much as the data.

That regularization suits a problem where prior knowledge rules out very large coefficients, and a report of the penalized fit should state \(\regpar\). What these six patients show is not that \(\hcoef{1} = 2.295\). They show that the dose separates the outcomes, that the evidence against \(\coef{1} = 0\) is modest (exact one-sided \(p = 0.05\), Solution 21), and that the data put no upper bound on the slope.

Theorem 1 (Separated data have no maximum likelihood estimate)

Theorem 1 (Separated data have no maximum likelihood estimate) Suppose \(\vcoef\) separates the data (Definition 1). Then \(\llik(c\vcoef) > \llik(\vcoef)\) for every \(c > 1\). The supremum of the log-likelihood \(\llik\) is \(0\), approached as \(c \to \infty\) along the ray \(c\vcoef\), and no finite parameter vector attains it, so the maximum likelihood estimate of the logistic regression does not exist.

Proof

Example 6 (The likelihood keeps rising along a ray)

Example 6 (The likelihood keeps rising along a ray) Consider the six patients of Exercise 9, with doses \(-3, -2, -1, 1, 2, 3\) and recoveries \(0, 0, 0, 1, 1, 1\). The vector \(\vcoef = \tp{(0, 1)}\) separates them (Example 1). Scale it by \(c = 1, 2, 5, 10\), which gives \(\coef{0} = 0\) and \(\coef{1} = c\).

Show code
sp_x <- c(-3, -2, -1, 1, 2, 3)
sp_y <- c(0, 0, 0, 1, 1, 1)
sp_llik <- function(b1) sum(dbinom(sp_y, 1, plogis(b1 * sp_x), log = TRUE))
sp_c <- c(1, 2, 5, 10)
data.frame(c = sp_c, loglik = vapply(sp_c, sp_llik, numeric(1)))

As Theorem 1 states, the log-likelihood increases with \(c\), from -0.978 at \(c = 1\) to -9.08e-05 at \(c = 10\). It approaches its supremum \(0\) and does not reach it, so the likelihood approaches its supremum \(1\) and does not reach it either.

Figure 5 plots the curve against \(c\).

Show code
library(ggplot2)
sep_grid <- data.frame(c = seq(0, 10, length.out = 300))
sep_grid$loglik <- vapply(sep_grid$c, sp_llik, numeric(1))
sep_marks <- data.frame(c = sp_c, loglik = vapply(sp_c, sp_llik, numeric(1)))
ggplot(sep_grid) +
  aes(c, loglik) +
  geom_hline(yintercept = 0, linetype = "dotted") +
  geom_line(colour = "#0072B2") +
  geom_point(data = sep_marks, colour = "#D55E00", shape = 17, size = 2.5) +
  labs(x = "Slope c", y = "Log-likelihood")
Show code
import matplotlib.pyplot as plt
import numpy as np

x_sep = np.array([-3, -2, -1, 1, 2, 3])
y_sep = np.array([0, 0, 0, 1, 1, 1])

def llik_sep(b1):
    p = 1 / (1 + np.exp(-b1 * x_sep))
    return np.sum(y_sep * np.log(p) + (1 - y_sep) * np.log(1 - p))

c_sep = np.linspace(0, 10, 300)
marks_sep = np.array([1, 2, 5, 10])
fig, ax = plt.subplots(figsize=(5, 4), layout="constrained")
ax.axhline(0, color="black", linestyle=":");
ax.plot(c_sep, [llik_sep(c) for c in c_sep], color="#0072B2");
ax.plot(marks_sep, [llik_sep(c) for c in marks_sep], color="#D55E00", marker="^", linestyle="");
ax.set_xlabel("Slope c");
ax.set_ylabel("Log-likelihood");
plt.show()
Figure 5: The log-likelihood of Example 6 at \(\coef{0} = 0\) and \(\coef{1} = c\) (solid), with the four values of \(c\) from the example marked. The curve rises toward its supremum \(0\) (dotted) and does not reach it.
(a)
Line plot of the log-likelihood of the six-patient logistic regression against the slope c from 0 to 10, with the intercept fixed at 0. The curve is negative, about -4.2 at c of 0 and rising steeply at first, then flattening as it approaches the dotted horizontal line at 0 without touching it. Four marked points show the slopes 1, 2, 5 and 10.
(b)
Line plot of the log-likelihood of the six-patient logistic regression against the slope c from 0 to 10, with the intercept fixed at 0. The curve is negative, about -4.2 at c of 0 and rising steeply at first, then flattening as it approaches the dotted horizontal line at 0 without touching it. Four marked points show the slopes 1, 2, 5 and 10.

Example 7 (Gradient descent on separated data has nowhere to stop)

Example 7 (Gradient descent on separated data has nowhere to stop) Table 2 lists four observations with one feature \(\predvar_i\), so \(\vx_i = \tp{(1, \predvar_i)}\) and \(\vcoef = \tp{(\coef{0}, \coef{1})}\).

Table 2: Four observations whose labels switch from 0 to 1 between \(\predvar = 2\) and \(\predvar = 4\).
\(i\) \(\predvar_i\) \(\outvar_i\)
1 1 0
2 2 0
3 4 1
4 5 1

The vector \(\vcoef = \tp{(-3, 1)}\) separates these observations: \(\dprod{\vcoef}{\vx_i} = \predvar_i - 3\) is negative for the two with \(\outvar_i = 0\) and positive for the two with \(\outvar_i = 1\). So, by Theorem 1, the total log loss \(C(\vcoef)\) has no minimizer. The log-likelihood \(\llik = -C\) is concave, so \(C\) is convex, and any point where the gradient \(\grad C(\vcoef)\) is \(\vzero\) would minimize \(C\). No point does, so \(\grad C(\vcoef)\) is never \(\vzero\), and gradient descent never reaches a point where it stops moving.

Figure 6 (a) draws the descent direction \(-\grad C(\vcoef)\) at a grid of points, on the contours of \(C\), and Figure 6 (b) draws the same arrows on its surface. By the logistic gradient theorem, \(\grad C(\vcoef) = \sumin (\prob_i - \outvar_i) \vx_i\), so

\[ \dprod{-\grad C(\vcoef)}{\tp{(-3, 1)}} = \sumin (\prob_i - \outvar_i) (3 - \predvar_i). \]

Each term is positive: an observation with \(\outvar_i = 0\) has \(\prob_i > 0\) and \(\predvar_i < 3\), and one with \(\outvar_i = 1\) has \(\prob_i < 1\) and \(\predvar_i > 3\). So every arrow has a part pointing along the dashed line \(\coef{0} + 3 \coef{1} = 0\) toward the upper left, where \(C\) keeps falling toward 0 without reaching it.

Figure 6: The descent direction \(-\grad C(\vcoef)\) (arrows, all drawn equally long) for the total log loss \(C\) of the data in Table 2. An arrow’s color shows the length \(\norm{\grad C(\vcoef)}\) of the gradient, on a logarithmic scale. The dashed line is \(\coef{0} + 3 \coef{1} = 0\), and every nonzero \(\vcoef\) on it has \(\dprod{\vcoef}{\vx} = 0\) exactly where \(\predvar = 3\).
Show code
sdd_x <- c(1, 2, 4, 5)
sdd_y <- c(0, 0, 1, 1)
sdd_obs <- tibble::tibble(x = sdd_x, y = sdd_y)
sdd_cost_grad <- tidyr::expand_grid(i = 0:100, j = 0:45) |>
  dplyr::mutate(
    b0 = -9 + i / 10,
    b1 = -1 + j / 10,
    on_arrow_grid = i %% 10 == 0 & j %% 10 == 0
  ) |>
  dplyr::cross_join(sdd_obs) |>
  dplyr::mutate(
    z = b0 + b1 * x,
    resid = plogis(z) - y,
    loss = -plogis(-z, log.p = TRUE) - y * z
  ) |>
  dplyr::summarise(
    cost = sum(loss),
    d0 = sum(resid),
    d1 = sum(resid * x),
    .by = c(b0, b1, on_arrow_grid)
  )
sdd_cost_range <- range(sdd_cost_grad$cost)
sdd_len <- 0.7
sdd_arrows <- sdd_cost_grad |>
  dplyr::filter(on_arrow_grid) |>
  dplyr::mutate(
    grad_norm = sqrt(d0^2 + d1^2),
    b0_end = b0 - sdd_len * d0 / grad_norm,
    b1_end = b1 - sdd_len * d1 / grad_norm
  )
sdd_head_size <- grid::unit(0.05, "inches")
sdd_head <- grid::arrow(length = sdd_head_size, type = "closed")
sdd_arrow_aes <- ggplot2::aes(
  x = b0, y = b1, xend = b0_end, yend = b1_end, colour = grad_norm
)
# Contours at values of C that double from 1/8 to 16 (see the text).
sdd_levels <- 2^(-3:4)
sdd_xlab <- expression(beta[0])
sdd_ylab <- expression(beta[1])
sdd_plot <- sdd_cost_grad |>
  ggplot2::ggplot() +
  ggplot2::aes(x = b0, y = b1, z = cost) +
  ggplot2::geom_contour(
    breaks = sdd_levels, # check-unexplained-choices: allow
    colour = "grey70"
  ) +
  ggplot2::geom_abline(intercept = 0, slope = -1 / 3, linetype = "dashed") +
  ggplot2::geom_segment(
    data = sdd_arrows,
    mapping = sdd_arrow_aes,
    inherit.aes = FALSE,
    arrow = sdd_head
  ) +
  ggplot2::scale_colour_viridis_c(transform = "log10") +
  ggplot2::coord_equal() +
  ggplot2::labs(x = sdd_xlab, y = sdd_ylab, colour = "gradient\nlength") +
  ggplot2::theme(legend.position = "bottom")
# The arrows' colours, reused by the surface.
sdd_layer <- ggplot2::layer_data(sdd_plot, 3)
sdd_arrows$colour <- sdd_layer$colour
sdd_plot
Show code
import matplotlib.pyplot as plt
import numpy as np
from matplotlib.colors import LogNorm

x_obs = np.array([1, 2, 4, 5])
y_obs = np.array([0, 0, 1, 1])


def cost_and_grad(b0, b1):
    z = b0[..., None] + b1[..., None] * x_obs
    resid = 1 / (1 + np.exp(-z)) - y_obs
    cost = (np.logaddexp(0, z) - y_obs * z).sum(axis=-1)
    return cost, resid.sum(axis=-1), (resid * x_obs).sum(axis=-1)


b0, b1 = np.meshgrid(np.linspace(-9, 1, 101), np.linspace(-1, 3.5, 46))
cost, _, _ = cost_and_grad(b0, b1)
a0, a1 = np.meshgrid(np.arange(-9, 2), np.arange(-1, 4))
_, d0, d1 = cost_and_grad(a0.astype(float), a1.astype(float))
norm = np.hypot(d0, d1)
fig, ax = plt.subplots(figsize=(4.5, 3.5), layout="constrained")
# Contours at values of C that double from 1/8 to 16 (see the text).
ax.contour(b0, b1, cost, levels=2.0 ** np.arange(-3, 5), colors="grey");
ax.axline((0, 0), slope=-1 / 3, color="black", linestyle="--");
arrows = ax.quiver(a0, a1, -d0 / norm, -d1 / norm, norm, norm=LogNorm(),
                   cmap="viridis", angles="xy", scale_units="xy", scale=1 / 0.7)
bar = fig.colorbar(arrows, ax=ax, location="bottom", label="gradient length")
bar.set_ticks([0.1, 0.3, 1, 3], labels=["0.1", "0.3", "1", "3"]);
bar.minorticks_off();
# Room for the arrows on the top and bottom rows.
ax.set_ylim(-1.3, 3.8);
ax.set_aspect("equal");
ax.set_xlabel(r"$\beta_0$");
ax.set_ylabel(r"$\beta_1$");
plt.show()
(a) Contours of \(C\) with the descent direction at each grid point.
Show code
sdd_grey_low <- c(0, "#e5e5e5")
sdd_grey_high <- c(1, "#808080")
sdd_grey <- list(sdd_grey_low, sdd_grey_high)
sdd_eye <- list(x = -1.4, y = -1.8, z = 1)
sdd_camera <- list(eye = sdd_eye)
sdd_beta <- intToUtf8(946)
sdd_xtitle <- paste0(sdd_beta, "<sub>0</sub>")
sdd_ytitle <- paste0(sdd_beta, "<sub>1</sub>")
sdd_xaxis <- list(title = sdd_xtitle)
sdd_yaxis <- list(title = sdd_ytitle)
sdd_zaxis <- list(title = "C")
sdd_scene <- list(
  xaxis = sdd_xaxis,
  yaxis = sdd_yaxis,
  zaxis = sdd_zaxis,
  aspectmode = "cube",
  camera = sdd_camera
)
sdd_widget <- plotly::plot_ly() |>
  plotly::add_surface(
    x = sdd_b0,
    y = sdd_b1,
    z = sdd_z,
    colorscale = sdd_grey,
    showscale = FALSE,
    opacity = 0.8
  )
# One trace per arrow: a shaft, and a marker at the tip.
sdd_n_arrows <- nrow(sdd_arrows)
for (k in seq_len(sdd_n_arrows)) {
  sdd_k <- sdd_arrows[k, ]
  sdd_line <- list(color = sdd_k$colour, width = 5)
  sdd_tip <- list(color = sdd_k$colour, size = 3)
  sdd_xs <- c(sdd_k$b0, sdd_k$b0_end)
  sdd_ys <- c(sdd_k$b1, sdd_k$b1_end)
  sdd_zs <- c(sdd_k$cost, sdd_k$cost_end)
  sdd_widget <- sdd_widget |>
    plotly::add_paths(
      x = sdd_xs, y = sdd_ys, z = sdd_zs, line = sdd_line,
      showlegend = FALSE, hoverinfo = "skip"
    ) |>
    plotly::add_markers(
      x = sdd_k$b0_end, y = sdd_k$b1_end, z = sdd_k$cost_end,
      marker = sdd_tip, showlegend = FALSE, hoverinfo = "skip"
    )
}
sdd_widget <- plotly::layout(sdd_widget, scene = sdd_scene)
htmltools::div(role = "img", `aria-label` = sdd_alt, sdd_widget)
Show code
import html
import matplotlib as mpl
import plotly.graph_objects as go

ends0, ends1 = a0 - 0.7 * d0 / norm, a1 - 0.7 * d1 / norm
start_cost, _, _ = cost_and_grad(a0.astype(float), a1.astype(float))
end_cost = start_cost - 0.7 * norm  # in the tangent plane at the start
colours = mpl.colormaps["viridis"](LogNorm()(norm))
fig = go.Figure(go.Surface(x=b0[0], y=b1[:, 0], z=cost, showscale=False,
                           opacity=0.8,
                           colorscale=[[0, "#e5e5e5"], [1, "#808080"]]))
for k in np.ndindex(norm.shape):
    colour = mpl.colors.to_hex(colours[k])
    fig.add_trace(go.Scatter3d(
        x=[a0[k], ends0[k]], y=[a1[k], ends1[k]], z=[start_cost[k], end_cost[k]],
        mode="lines", line=dict(color=colour, width=5), showlegend=False,
        hoverinfo="skip"));
    fig.add_trace(go.Scatter3d(
        x=[ends0[k]], y=[ends1[k]], z=[end_cost[k]], mode="markers",
        marker=dict(color=colour, size=3), showlegend=False,
        hoverinfo="skip"));
beta = chr(946)  # Greek beta, as a code point to keep the source ASCII
fig.update_layout(scene=dict(xaxis_title=beta + "<sub>0</sub>",
                             yaxis_title=beta + "<sub>1</sub>",
                             zaxis_title="C", aspectmode="cube",
                             camera=dict(eye=dict(x=-1.4, y=-1.8, z=1))));
print("```{=html}\n"
      + f'<div role="img" aria-label="{html.escape(r.sdd_alt)}">\n'
      + fig.to_html(full_html=False, include_plotlyjs="cdn") + "\n</div>\n```")
(b) Surface of \(C\) with the same arrows, each drawn in the tangent plane where it starts. Drag the surface to rotate it.

Remark 2 (Why the contour levels double)

Remark 2 (Why the contour levels double). The contours in Figure 6 (a) are drawn at values of \(C\) that double from 1/8 to 16, because over the plotted window \(C\) runs from 0.096, slightly above the dashed line at the upper left, to 27, at the lower left. Evenly spaced levels would crowd into the lower left and leave none where the arrows head.

Example 8 (The zero curves of the two partial derivatives never meet)

Example 8 (The zero curves of the two partial derivatives never meet) For the four observations of Table 2, the gradient \(\grad C(\vcoef)\) of the total log loss is \(\vzero\) only where both partial derivatives are 0, so such a point would lie on the zero curve of each. Example 7 showed that

\[ 3 \derivf{C}{\coef{0}} - \derivf{C}{\coef{1}} = \sumin (\prob_i - \outvar_i) (3 - \predvar_i) > 0 \]

for every \(\vcoef\). So wherever \(\derivf{C}{\coef{0}} = 0\), \(\derivf{C}{\coef{1}}\) is negative, and the two zero curves never meet.

Figure 7 (a) draws each partial derivative as level curves and Figure 7 (b) as a surface, with both zero curves in every panel. The zero curve of \(\derivf{C}{\coef{0}}\) (solid) is the line \(\coef{0} + 3 \coef{1} = 0\), because \(\derivf{C}{\coef{0}} = \sumin (\prob_i - \outvar_i)\) and on that line the four predicted probabilities are \(\expitf{-2 \coef{1}}\), \(\expitf{-\coef{1}}\), \(\expitf{\coef{1}}\) and \(\expitf{2 \coef{1}}\), which pair up as \(\expitf{-t} + \expitf{t} = 1\) and so add up to 2, the number of observations with \(\outvar_i = 1\). Off the line, \(\derivf{C}{\coef{0}}\) is not 0, because it increases with \(\coef{0}\) for each fixed \(\coef{1}\). The zero curve of \(\derivf{C}{\coef{1}}\) (dotted) lies above the solid curve: on the solid curve \(\derivf{C}{\coef{1}}\) is negative, for the reason given at the start of this example, and \(\derivf{C}{\coef{1}}\) increases with \(\coef{1}\) for each fixed \(\coef{0}\), because the partial derivative of \(\derivf{C}{\coef{1}}\) in \(\coef{1}\) is \(\sumin \prob_i (1 - \prob_i) \predvar_i^2 > 0\).

Figure 7: The partial derivatives of the total log loss \(C\) for the data in Table 2, colored from blue (negative) through gray to red (positive). The solid black curve is where \(\derivf{C}{\coef{0}} = 0\), and the dotted black curve is where \(\derivf{C}{\coef{1}} = 0\). The solid curve is the dashed line \(\coef{0} + 3 \coef{1} = 0\) of Figure 6.
Show code
spd_long <- sdd_cost_grad |>
  dplyr::select(b0, b1, d0, d1) |>
  tidyr::pivot_longer(
    cols = dplyr::starts_with("d"),
    names_to = "partial",
    values_to = "value"
  ) |>
  dplyr::mutate(
    panel = dplyr::if_else(
      partial == "d0",
      "partialdiff * C / partialdiff * beta[0]",
      "partialdiff * C / partialdiff * beta[1]"
    )
  )
# Level curves at these values of each partial derivative (see the text).
spd_levels <- c(-8, -4, -2, -1, -0.5, 0.5, 1, 2)
spd_level_aes <- ggplot2::aes(colour = ggplot2::after_stat(level))
spd_zero0_aes <- ggplot2::aes(x = b0, y = b1, z = d0, linetype = "d0")
spd_zero1_aes <- ggplot2::aes(x = b0, y = b1, z = d1, linetype = "d1")
spd_zero_labels <- expression(
  partialdiff * C / partialdiff * beta[0] == 0,
  partialdiff * C / partialdiff * beta[1] == 0
)
spd_zero_types <- c(d0 = "solid", d1 = "dotted")
spd_panel <- ggplot2::vars(panel)
spd_plot <- spd_long |>
  ggplot2::ggplot() +
  ggplot2::aes(x = b0, y = b1, z = value) +
  ggplot2::geom_contour(
    mapping = spd_level_aes,
    breaks = spd_levels # check-unexplained-choices: allow
  ) +
  ggplot2::geom_contour(
    data = sdd_cost_grad, mapping = spd_zero0_aes,
    breaks = 0, colour = "black" # check-unexplained-choices: allow
  ) +
  ggplot2::geom_contour(
    data = sdd_cost_grad, mapping = spd_zero1_aes,
    breaks = 0, colour = "black" # check-unexplained-choices: allow
  ) +
  ggplot2::facet_wrap(spd_panel, ncol = 1, labeller = ggplot2::label_parsed) +
  ggplot2::scale_colour_gradient2(
    low = "#2166ac", mid = "grey85", high = "#b2182b", midpoint = 0
  ) +
  ggplot2::scale_linetype_manual(
    values = spd_zero_types, labels = spd_zero_labels
  ) +
  ggplot2::coord_equal() +
  ggplot2::labs(
    x = sdd_xlab, y = sdd_ylab, colour = "level", linetype = NULL
  ) +
  ggplot2::theme(legend.position = "bottom", legend.box = "vertical")
spd_plot
Show code
_, d0_grid, d1_grid = cost_and_grad(b0, b1)
# Level curves at these values of each partial derivative (see the text).
levels = [-8, -4, -2, -1, -0.5, 0.5, 1, 2]
fig, axes = plt.subplots(2, 1, figsize=(4.5, 5.5), layout="constrained",
                         sharex=True)
for ax, z, name in zip(axes, [d0_grid, d1_grid], ["0", "1"]):
    ax.contour(b0, b1, z, levels=levels, cmap="RdBu_r", vmin=-8, vmax=8);
    ax.contour(b0, b1, d0_grid, levels=[0], colors="black");
    ax.contour(b0, b1, d1_grid, levels=[0], colors="black",
               linestyles="dotted");
    ax.set_aspect("equal");
    ax.set_title(rf"$\partial C / \partial \beta_{name}$");
    ax.set_ylabel(r"$\beta_1$");
axes[1].set_xlabel(r"$\beta_0$");
plt.show()
(a) Level curves of each partial derivative, with both zero curves.
Show code
spd_alt <- paste(
  "Two interactive 3D surfaces over the plane of the intercept and the",
  "slope: the partial derivatives of the total log loss in the intercept",
  "and in the slope, red where positive and blue where negative.",
  "Each crosses height 0 along its own black zero curve,",
  "solid for the intercept and dotted for the slope,",
  "and the two curves never meet. Buttons show one surface at a time."
)
spd_scale_low <- c(0, "#2166ac")
spd_scale_mid <- c(0.5, "#d9d9d9")
spd_scale_high <- c(1, "#b2182b")
spd_scale <- list(spd_scale_low, spd_scale_mid, spd_scale_high)
spd_names <- c("dC/d(beta 0)", "dC/d(beta 1)")
spd_surfaces <- list(spd_d0, spd_d1)
spd_widget <- plotly::plot_ly()
for (k in 1:2) {
  spd_widget <- spd_widget |>
    plotly::add_surface(
      x = sdd_b0, y = sdd_b1, z = spd_surfaces[[k]], name = spd_names[k],
      colorscale = spd_scale, cmin = -8, cmax = 8, showscale = FALSE,
      opacity = 0.85
    )
}
spd_dashes <- c("solid", "dot")
for (k in 1:2) {
  spd_line <- list(color = "black", width = 5, dash = spd_dashes[k])
  for (piece in spd_zero_sets[[k]]) {
    spd_widget <- spd_widget |>
      plotly::add_paths(
        x = piece$x, y = piece$y, z = 0 * piece$x, line = spd_line,
        showlegend = FALSE, hoverinfo = "skip"
      )
  }
}
# Buttons show both surfaces or one; the zero curves stay visible.
spd_built <- plotly::plotly_build(spd_widget)
spd_n_traces <- length(spd_built$x$data)
spd_curves <- rep(TRUE, spd_n_traces - 2)
spd_show_both <- c(TRUE, TRUE)
spd_show_d0 <- c(TRUE, FALSE)
spd_show_d1 <- c(FALSE, TRUE)
spd_show <- list(spd_show_both, spd_show_d0, spd_show_d1)
spd_labels <- c("both", spd_names)
spd_buttons <- list()
for (k in 1:3) {
  spd_shown <- spd_show[[k]]
  spd_visible <- c(spd_shown, spd_curves)
  spd_restyle <- list(visible = spd_visible)
  spd_args <- list(spd_restyle)
  spd_buttons[[k]] <- list(
    method = "restyle", args = spd_args, label = spd_labels[k]
  )
}
spd_menu <- list(type = "buttons", direction = "right", buttons = spd_buttons)
spd_menus <- list(spd_menu)
spd_zaxis <- list(title = "partial derivative")
spd_scene <- list(
  xaxis = sdd_xaxis, yaxis = sdd_yaxis, zaxis = spd_zaxis,
  aspectmode = "cube", camera = sdd_camera
)
spd_widget <- plotly::layout(
  spd_widget, scene = spd_scene, updatemenus = spd_menus
)
htmltools::div(role = "img", `aria-label` = spd_alt, spd_widget)
Show code
scale = [[0, "#2166ac"], [0.5, "#d9d9d9"], [1, "#b2182b"]]
fig = go.Figure()
for z, name in [(d0_grid, "dC/d(beta 0)"), (d1_grid, "dC/d(beta 1)")]:
    fig.add_trace(go.Surface(x=b0[0], y=b1[:, 0], z=z, name=name,
                             colorscale=scale, cmin=-8, cmax=8,
                             showscale=False, opacity=0.85));
for z, dash in [(d0_grid, "solid"), (d1_grid, "dot")]:
    zero = plt.contour(b0, b1, z, levels=[0])
    for path in zero.allsegs[0]:
        fig.add_trace(go.Scatter3d(x=path[:, 0], y=path[:, 1], z=0 * path[:, 0],
                                   mode="lines", showlegend=False,
                                   line=dict(color="black", width=5, dash=dash)));
plt.close()
curves = [True] * (len(fig.data) - 2)
buttons = [dict(method="restyle", label=label, args=[{"visible": show + curves}])
           for label, show in [("both", [True, True]),
                               ("dC/d(beta 0)", [True, False]),
                               ("dC/d(beta 1)", [False, True])]]
fig.update_layout(
    updatemenus=[dict(type="buttons", direction="right", buttons=buttons)],
    scene=dict(xaxis_title=beta + "<sub>0</sub>",
               yaxis_title=beta + "<sub>1</sub>",
               zaxis_title="partial derivative", aspectmode="cube",
               camera=dict(eye=dict(x=-1.4, y=-1.8, z=1))));
print("```{=html}\n"
      + f'<div role="img" aria-label="{html.escape(r.spd_alt)}">\n'
      + fig.to_html(full_html=False, include_plotlyjs="cdn") + "\n</div>\n```")
(b) Each partial derivative as a surface, with both zero curves at height 0. Drag to rotate; the buttons show one surface at a time.

Remark 3 (Why these level curves)

Remark 3 (Why these level curves). The level curves in Figure 7 (a) are drawn at \(\pm 0.5\), \(\pm 1\), \(\pm 2\), \(-4\) and \(-8\), because over the plotted window \(\derivf{C}{\coef{0}}\) runs from -2 to 2 and \(\derivf{C}{\coef{1}}\) from -9 to 3. Levels that double away from 0 cover both of them with the same colors, and leave room near 0 for the zero curves.

Show code
library(ggplot2)

set.seed(571)
n_per_class <- 15
sim_df <- data.frame(
  x1 = c(rnorm(n_per_class, -2.5, 0.6), rnorm(n_per_class, 2.5, 0.6)),
  x2 = rnorm(2 * n_per_class, 0, 0.9),
  class = rep(c("Left", "Right"), each = n_per_class)
)

# Each boundary passes through the midpoint of the class means,
# at a different angle (in degrees) from the horizontal axis.
pivot <- c(mean(sim_df$x1), mean(sim_df$x2))
angles <- c(f1 = 55, f2 = 90, f3 = 125)
boundary_df <- data.frame(
  boundary = paste(names(angles), "= 0"),
  angle = angles * pi / 180
)

# Check that every boundary separates the two classes.
side <- function(angle) {
  (sim_df$x1 - pivot[1]) * sin(angle) - (sim_df$x2 - pivot[2]) * cos(angle) > 0
}
stopifnot(all(vapply(
  boundary_df$angle,
  function(a) nrow(unique(data.frame(side(a), sim_df$class))) == 2,
  logical(1)
)))

# geom_abline cannot draw a vertical line, so draw each boundary as a segment.
boundary_df$x_start <- pivot[1] - 10 * cos(boundary_df$angle)
boundary_df$x_end <- pivot[1] + 10 * cos(boundary_df$angle)
boundary_df$y_start <- pivot[2] - 10 * sin(boundary_df$angle)
boundary_df$y_end <- pivot[2] + 10 * sin(boundary_df$angle)

ggplot(sim_df) +
  aes(x1, x2) +
  geom_point(aes(colour = class, shape = class), size = 2.5) +
  geom_segment(
    data = boundary_df,
    aes(x = x_start, y = y_start, xend = x_end, yend = y_end, linetype = boundary)
  ) +
  scale_colour_manual(values = c(Left = "#D55E00", Right = "#0072B2")) +
  scale_shape_manual(values = c(Left = 3, Right = 16)) +
  scale_linetype_manual(values = c(
    "f1 = 0" = "solid", "f2 = 0" = "dashed", "f3 = 0" = "dotted"
  )) +
  coord_cartesian(xlim = range(sim_df$x1), ylim = range(sim_df$x2)) +
  labs(x = "Feature 1, x1", y = "Feature 2, x2", colour = NULL, shape = NULL, linetype = NULL)
Show code
import matplotlib.pyplot as plt
import numpy as np

rng = np.random.default_rng(571)
n_per_class = 15
left = np.column_stack([rng.normal(-2.5, 0.6, n_per_class),
                        rng.normal(0, 0.9, n_per_class)])
right = np.column_stack([rng.normal(2.5, 0.6, n_per_class),
                         rng.normal(0, 0.9, n_per_class)])
points = np.vstack([left, right])
is_right = np.repeat([False, True], n_per_class)

# Each boundary passes through the midpoint of the class means,
# at a different angle (in degrees) from the horizontal axis.
pivot = points.mean(axis=0)
angles = {"f1": 55, "f2": 90, "f3": 125}
styles = {"f1": "-", "f2": "--", "f3": ":"}


def side(angle):
    offset = points - pivot
    return offset[:, 0] * np.sin(angle) - offset[:, 1] * np.cos(angle) > 0


# Check that every boundary separates the two classes.
for degrees in angles.values():
    assert len(set(zip(side(np.radians(degrees)), is_right))) == 2

fig, ax = plt.subplots(figsize=(5, 3.6), layout="constrained")
ax.scatter(*left.T, marker="+", color="#D55E00", label="Left");
ax.scatter(*right.T, marker="o", color="#0072B2", label="Right");
x_lim, y_lim = ax.get_xlim(), ax.get_ylim()
for name, degrees in angles.items():
    direction = np.array([np.cos(np.radians(degrees)), np.sin(np.radians(degrees))])
    ends = pivot + np.outer([-10, 10], direction)
    ax.plot(*ends.T, styles[name], color="black", label=f"{name} = 0");
ax.set_xlim(x_lim);
ax.set_ylim(y_lim);
ax.set_xlabel("Feature 1, x1");
ax.set_ylabel("Feature 2, x2");
fig.legend(loc="outside lower center", ncol=5, frameon=False);
plt.show()
Figure 8: Three linear (strictly, affine) boundaries \(f(\vx) = 0\), where \(f(\vx) = \coef{0} + \coef{1} x_1 + \coef{2} x_2\), that all separate the same two classes.
(a)
A scatter plot of two simulated clusters of points, one on the left and one on the right, with a gap between them. Three straight lines, drawn solid, dashed and dotted and labelled f1 = 0, f2 = 0 and f3 = 0, cross at a single point in the gap. Each line has the left cluster on one side and the right cluster on the other, but the lines are at different distances from the nearest points.
(b)
A scatter plot of two simulated clusters of points, one on the left and one on the right, with a gap between them. Three straight lines, drawn solid, dashed and dotted and labelled f1 = 0, f2 = 0 and f3 = 0, cross at a single point in the gap. Each line has the left cluster on one side and the right cluster on the other, but the lines are at different distances from the nearest points.

Every boundary in Figure 8 classifies the simulated points without error, and the one farthest from the nearest point of either class has the largest margin. Maximizing the logistic likelihood does not give that boundary. The log loss of an observation, \(\log\paren{1 + \e{-m}}\), decreases exponentially in its margin \(m = \signoutvar \dprod{\vcoef}{\vx}\) and never reaches \(0\), so on separated data the likelihood keeps increasing as \(\norm{\vcoef}\) grows along a separating direction, and there is no finite maximizer (Theorem 1). A ridge penalty gives a finite maximizer (Solution 22). As the penalty weight tends to \(0\), the direction of the penalized fit approaches that of the maximum-margin boundary [6]. Support vector machines maximize the margin of the separating hyperplane directly, and their solution is finite.

4 Separation in real data

The six patients of Exercise 9 were made up so the algebra could be done by hand. Separation also turns up in published studies, and fitting software reports it no more clearly there.

Dataset 2: The endometrial cancer data

Heinze and Schemper analyzed endometrial cancer data on 79 patients as an example of separation [7]. Asseryanis, of the Medical University of Vienna, provided the data. The copy here comes from the endometrial table in the R package brglm2. The data are in data/endometrial.csv, one patient per line:

  • HG: the outcome, the histology grade of the tumor, 1 for high grade and 0 for low grade.
  • NV: neovascularization, 1 if new blood vessels have formed in the tumor and 0 if not.
  • PI: the pulsatility index of the uterine artery.
  • EH: the height of the endometrium.
Dataset 2: The endometrial cancer data

Exercise 25 (Separation in the endometrial cancer data: what software reports)

Exercise 25 (Separation in the endometrial cancer data: what software reports) Data from published studies can be quasi-completely separated too. In the endometrial cancer data, tabulating the histology grade HG against neovascularization NV shows that every patient with NV \(= 1\) has a high-grade tumor:

Show code
# Read the local copy, then count the patients in each NV and HG cell,
# listing the empty cell with a count of 0 rather than dropping it.
endo <- readr::read_csv("data/endometrial.csv", show_col_types = FALSE)
empty_cell_count <- list(n = 0L)
endo |>
  dplyr::count(NV, HG) |>
  tidyr::complete(NV, HG, fill = empty_cell_count)

Fit the logistic regression of HG on NV with R’s glm(). What does it report?

Hint

Compare the report with the quasi-completely separated example of Example 2.

Solution 25. Fitting the endometrial cancer data with R’s glm():

Show code
# Fit HG on NV, then tabulate the coefficients with parameters.
endo_fit <- glm(HG ~ NV, family = binomial, data = endo)
endo_coefs <- summary(endo_fit)$coefficients
endo_b1_hat <- coef(endo_fit)[["NV"]]
endo_se_hat <- endo_coefs["NV", "Std. Error"]
endo_p_hat <- endo_coefs["NV", "Pr(>|z|)"]
parameters::parameters(endo_fit) |> parameters::print_md(digits = 3)
Parameter Log-Odds SE 95% CI z p
(Intercept) -1.059 0.281 (-1.639, -0.528) -3.761 < .001
NV 19.625 1809.054 (-104.915, ) 0.011 0.991

R reports a slope of 19.6 for NV, with a standard error of 1809 and a \(p\)-value of 0.991. It gives no warning. An odds ratio of \(\expf{19.6}\) is not a plausible value for any predictor.

NV quasi-completely separates the data. The 13 patients with NV \(= 1\) all have HG \(= 1\), while the 66 patients with NV \(= 0\) include both grades. So the maximum likelihood estimate of the slope does not exist [4], [7], and the only sign of trouble in the output is the large standard error.

Exercise 26 (Separation in the endometrial cancer data: why the fit stops)

Exercise 26 (Separation in the endometrial cancer data: why the fit stops) R’s glm() reports a slope for NV when it fits the logistic regression of the histology grade HG on NV in the endometrial cancer data (Solution 25).

Why does glm() stop at that slope?

Hint

Compare the stopped fit for the six patients of Exercise 9 in Example 4.

Solution 26. The log-likelihood of the endometrial cancer data keeps increasing as the slope of NV grows. Each increase moves the fitted probability for the patients with NV \(= 1\) closer to their observed \(1\). Far out, the log-likelihood changes very little. There the gradient and the change in fit are both small, so a fitting routine stops there, at a slope that its tolerance determines, without having found a maximum. glm() stops after 17 iterations, when the relative change in the deviance falls below its tolerance, epsilon = 1e-8 by default. The reported slope \(19.62\) depends on that tolerance; it is not an estimate of a finite parameter. Running the same fit with a tighter tolerance:

Show code
# Tighten glm's convergence tolerance and allow more iterations.
endo_control <- glm.control(epsilon = 1e-14, maxit = 100)
endo_tight <- glm(
  HG ~ NV,
  family = binomial,
  data = endo,
  control = endo_control
)
c(slope = coef(endo_tight)[["NV"]], iterations = endo_tight$iter)
     slope iterations 
  32.62489   30.00000 

returns a larger slope, \(32.62\), after 30 iterations. Only this fit makes R warn, “fitted probabilities numerically 0 or 1 occurred”. At the default tolerance the fit stopped before any fitted probability rounded to \(1\).


Exercise 27 (Separation in the endometrial cancer data: the likelihood ratio test)

Exercise 27 (Separation in the endometrial cancer data: the likelihood ratio test) R’s glm() gives the slope of NV a Wald test \(p\)-value of 0.991 in the logistic regression of the histology grade HG on NV in the endometrial cancer data (Solution 25).

Compute the likelihood ratio statistic \(G^2\) for \(H_0\): the slope of NV is \(0\), and its \(p\)-value from the \(\chi^2_1\) distribution.

Hint

\(G^2\) is the null deviance minus the deviance of the fit, as in Solution 19. In R, anova() with test = "LRT" compares two nested fits.

Solution 27. Under \(H_0\) the slope of NV is \(0\), so the null fit has an intercept alone. The unrestricted fit has no maximizer, so the log-likelihood at the slope where glm() stopped takes the place of the maximized log-likelihood in the likelihood ratio statistic. That log-likelihood is less than \(10^{-6}\) below its supremum, because the log-likelihood changes very little that far out (Solution 26). Comparing the two fits of the endometrial cancer data:

Show code
# Fit the intercept-only model, then compare it with the fit on NV.
endo_null <- glm(HG ~ 1, family = binomial, data = endo)
endo_lrt <- anova(endo_null, endo_fit, test = "LRT")
endo_g2 <- endo_lrt$Deviance[[2]]
endo_g2_p <- endo_lrt[["Pr(>Chi)"]][[2]]
endo_lrt

\(G^2\) is the null deviance minus the deviance of the fit:

\[ \begin{aligned} G^2 &= 104.9 - 75.31 \\ &= 29.6. \end{aligned} \]

Against the \(\chi^2_1\) distribution, \(G^2\) gives \(p = 5.3\times 10^{-8}\), while the Wald test of the same hypothesis gave \(p = 0.991\).

Exercise 28 (Separation in the endometrial cancer data: which test to use)

Exercise 28 (Separation in the endometrial cancer data: which test to use) For the slope of NV in the logistic regression of the histology grade HG on NV in the endometrial cancer data, the Wald test gives \(p = 0.991\) (Solution 25) and the likelihood ratio test gives \(p = 5.3\times 10^{-8}\) (Solution 27).

Which of the two should you use here?

Hint

Look again at the Hauck–Donner effect.

Solution 28. Use the likelihood ratio test: it finds a strong association between NV and the histology grade HG. The Wald statistic is the slope divided by its standard error, and glm() stopped where both are large and the standard error has grown faster than the slope, so the Wald statistic is near \(0\). That decrease is the Hauck–Donner effect. The Wald test uses the curvature of the log-likelihood at the fitted slope, far out, where the log-likelihood changes very little. The likelihood ratio test uses the change in log-likelihood between a slope of \(0\) and the fit, which shows how much better the quasi-completely separated fit describes the endometrial cancer data.

Exercise 29 (Separation in the endometrial cancer data: how far to trust the p-value)

Exercise 29 (Separation in the endometrial cancer data: how far to trust the p-value) For the slope of NV in the logistic regression of the histology grade HG on NV in the endometrial cancer data, the likelihood ratio test \(p\)-value of Solution 27 relies on the \(\chi^2_1\) approximation. Only 13 patients have NV \(= 1\).

Compute a \(p\)-value that needs no approximation, by conditioning on the number of high-grade tumors.

Hint

With one binary predictor, conditioning on the number of patients with NV \(= 1\) as well as the number of high-grade tumors gives Fisher’s exact test, fisher.test() in R. Compare the six patients’ test in Solution 21.

Solution 29. The endometrial cancer data have 30 high-grade tumors among 79 patients. Under \(H_0\), given that count and the 13 patients with NV \(= 1\), each way to choose which patients have high-grade tumors is equally likely. The \(p\)-value is the probability of a table at least as extreme as the observed one, which is what Fisher’s exact test computes:

Show code
# Condition on the row and column totals of the NV by HG table.
endo_exact <- fisher.test(endo_tab)
endo_exact$p.value
[1] 4.537619e-07

The two-sided \(p\)-value from fisher.test() is \(4.5\times 10^{-7}\). It is larger than the \(\chi^2_1\) value of Solution 27, \(5.3\times 10^{-8}\), but both reject \(H_0\) at any usual level, while the Wald test gave 0.991.

Exercise 30 (Separation in the endometrial cancer data: the penalized slope)

Exercise 30 (Separation in the endometrial cancer data: the penalized slope) In the logistic regression of the histology grade HG on NV in the endometrial cancer data, let \(\coef{0}\) be the intercept, \(\coef{1}\) the slope of NV, and \(\llik(\coef{0}, \coef{1})\) the log-likelihood. The ridge-penalized log-likelihood penalizes the slope alone:

\[ f(\coef{0}, \coef{1}) = \llik(\coef{0}, \coef{1}) - \tfrac{\regpar}{2}\coef{1}^2. \]

Find the slope \(\hcoef{1}\) of the maximizer of \(f\) for \(\regpar = 1\), \(0.1\) and \(0.01\).

Hint

Maximize \(f\) numerically, for example with R’s optim() on \(-f\). Compare the six patients’ penalized slopes in Solution 23.

Solution 30. Maximizing \(f\) numerically for the endometrial cancer data, from a start at \(\coef{0} = \coef{1} = 0\):

Show code
# Minus the ridge-penalized log-likelihood.
# The penalty leaves the intercept free.
endo_neg_pen <- function(b, lambda, data) { # nolint: function_location_linter
  eta <- b[[1]] + b[[2]] * data$NV
  p <- plogis(eta)
  loglik_i <- dbinom(data$HG, size = 1, prob = p, log = TRUE)
  loglik <- sum(loglik_i)
  lambda / 2 * b[[2]]^2 - loglik
}
# A tight tolerance, so the reported slopes do not depend on optim's default.
endo_optim_control <- list(reltol = 1e-14)
endo_ridge_slope <- function(lambda, data) { # nolint: function_location_linter
  start <- c(0, 0)
  fit <- optim(
    start, endo_neg_pen,
    lambda = lambda, data = data,
    method = "BFGS", control = endo_optim_control
  )
  fit$par[[2]]
}
endo_lambda <- c(1, 0.1, 0.01)
endo_ridge <- endo_lambda |> purrr::map_dbl(endo_ridge_slope, data = endo)
endo_ridge_cols <- c("$\\regpar$", "$\\hcoef{1}$")
tibble::tibble(lambda = endo_lambda, slope = endo_ridge) |>
  knitr::kable(
    col.names = endo_ridge_cols,
    digits = 3,
    align = "r"
  )
\(\regpar\) \(\hcoef{1}\)
1.00 2.376
0.10 4.380
0.01 6.368

The penalized slope is \(2.38\) at \(\regpar = 1\) and grows to \(6.37\) at \(\regpar = 0.01\).

Exercise 31 (Separation in the endometrial cancer data: what the penalty costs)

Exercise 31 (Separation in the endometrial cancer data: what the penalty costs) The ridge-penalized fits of the endometrial cancer data in Solution 30 give a slope for NV at each of \(\regpar = 1\), \(0.1\) and \(0.01\).

What does the penalty achieve, and at what cost?

Hint

Compare the six patients’ answer in Solution 24.

Solution 31. For the endometrial cancer data, the ridge penalty gives \(f\) a maximizer at every \(\regpar > 0\), so the slope for NV is a number that the fit can report.

The cost is that the slope depends on \(\regpar\). It is 2.7 times larger at \(\regpar = 0.01\) than at \(\regpar = 1\), and it keeps growing as \(\regpar \to 0\), because the likelihood alone keeps increasing as the slope grows. The penalty adds a preference for smaller coefficients, not information about NV from the data, and each penalized slope reflects that preference as much as the data. A report of a penalized fit should therefore state \(\regpar\).

What the 79 patients show is not that \(\hcoef{1} = 4.38\). They show that NV quasi-completely separates the histology grades, that a slope of \(0\) is implausible (\(p = 4.5\times 10^{-7}\), Solution 29), and that the data alone cannot say how large the slope is.

[1]
G. James, D. Witten, T. Hastie, R. Tibshirani, and J. Taylor, An introduction to statistical learning: With applications in Python. Springer, 2023. doi: 10.1007/978-3-031-38747-0.
[2]
A. Galanos, rugarch: Univariate GARCH models. 2026. doi: 10.32614/CRAN.package.rugarch.
[3]
T. Hastie, R. Tibshirani, and J. Taylor, “Statistical learning with python.” Stanford Online, lecture videos on YouTube, 2023. Available: https://www.youtube.com/playlist?list=PLoROMvodv4rPP6braWoRt5UCXYZ71GZIQ
[4]
A. Albert and J. A. Anderson, “On the existence of maximum likelihood estimates in logistic regression models,” Biometrika, vol. 71, no. 1, pp. 1–10, 1984, doi: 10.1093/biomet/71.1.1.
[5]
Jr. Hauck Walter W. and A. Donner, “Wald’s test as applied to hypotheses in logit analysis,” Journal of the American Statistical Association, vol. 72, no. 360, pp. 851–853, 1977, doi: 10.1080/01621459.1977.10479969.
[6]
S. Rosset, J. Zhu, and T. Hastie, “Boosting as a regularized path to a maximum margin classifier,” Journal of Machine Learning Research, vol. 5, pp. 941–973, 2004, Available: https://www.jmlr.org/papers/volume5/rosset04a/rosset04a.pdf
[7]
G. Heinze and M. Schemper, “A solution to the problem of separation in logistic regression,” Statistics in Medicine, vol. 21, no. 16, pp. 2409–2419, 2002, doi: 10.1002/sim.1047.