Definition 1 (Poisson regression model) Let \(Y\) be a count outcome, \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}\) a covariate vector, and \(T\) the exposure magnitude over which the events in \(Y\) are counted. A Poisson regression model assumes that, given \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{X}}\vphantom{X}}= \mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and \(T = t\), \(Y\) has a Poisson distribution with mean\(\mu(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}, t) = \lambda(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}) \cdot t\), where the event rate\(\lambda(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})\) has a log that is a linear function of the covariates:
Definition 2 (Rate ratio) In a Poisson regression model, the rate ratio comparing covariate pattern\(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) to covariate pattern \(\mathord{{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}^*}}\) is the ratio of their event rates:
Theorem 1 (Rate ratios in Poisson regression) In a Poisson regression model, suppose covariate\(x_j\) enters the linear function\({\ln\mathopen{}\left(\lambda(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})\right)\mathclose{}}\) only through the single term\(\beta_j x_j\) (no interactions or transformations involving \(x_j\)). Let \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and \(\mathord{{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}^*}}\) be two covariate patterns that are identical except that \(x_j = a\) and \(x^*_j = b\). Then
\[\text{ϙ}_{{\lambda}}(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}: \mathord{{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}^*}}) = {\operatorname{exp}\mathopen{}\left\{\beta_j (a - b)\right\}\mathclose{}} \tag{1}\]
Proof. Because \(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}\) and \(\mathord{{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}^*}}\) agree on every covariate except \(x_j\), every term of \({\ln\mathopen{}\left(\lambda(\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}})\right)\mathclose{}}\) except \(\beta_j x_j\) is shared with \({\ln\mathopen{}\left(\lambda(\mathord{{\mathord{\smash[b]{\underset{\raise.3ex\hbox{$\scriptstyle\sim$}}{x}}\vphantom{x}}^*}})\right)\mathclose{}}\):
ExmExample 1 (Rate ratio for a ten-year age difference)
Example 1 (Rate ratio for a ten-year age difference)
Suppose a Poisson regression model for emergency-room visits includes age in years, \(x_1\), with coefficient \(\beta_1 = 0.03\) and no interactions involving age. By Theorem 1 with \(a = 60\) and \(b = 50\), the rate ratio comparing a 60-year-old to a 50-year-old with the same values of every other covariate is:
A one-year age difference instead gives \({\operatorname{exp}\mathopen{}\left\{0.03\right\}\mathclose{}} \approx 1.030\), a 3% higher event rate per year of age. The ten-year rate ratio is the one-year rate ratio multiplied by itself ten times, \({\operatorname{exp}\mathopen{}\left\{0.3\right\}\mathclose{}} = \mathopen{}\left({\operatorname{exp}\mathopen{}\left\{0.03\right\}\mathclose{}}\right)\mathclose{}^{10}\), because rate ratios multiply across successive one-year steps.
3 Lab: Poisson regression on bike rentals
Hourly counts of bikes rented in a bike-sharing program in Washington, D.C., on 8645 hours make up the Bikeshare data [1, Sec. 4.6.2]. The data are in data/bikeshare.csv, one line per hour, copied from the book’s ISLR2 R package. The exercises use these variables:
bikers: the outcome, the number of bikes rented in that hour.
temp: the temperature, scaled to run from \(0\) to \(1\).
workingday: 1 on a working day and 0 otherwise.
weathersit: the weather, one of clear, cloudy/misty, light rain/snow, and heavy rain/snow.
# The first rows of the four columns that the exercises use.bikeshare_raw|>dplyr::select(bikers, temp, workingday, weathersit)|>head()
Show code
import pandas as pd# The first rows of the four columns that the exercises use.bikeshare_raw = pd.read_csv("data/bikeshare.csv")bikeshare_raw[["bikers", "temp", "workingday", "weathersit"]].head(6)#> bikers temp workingday weathersit#> 0 16 0.24 0 clear#> 1 40 0.22 0 clear#> 2 32 0.22 0 clear#> 3 13 0.24 0 clear#> 4 1 0.24 0 clear#> 5 1 0.24 0 cloudy/misty
The exercises follow the Poisson regression part of the book’s Chapter 4 lab [1, Sec. 4.7.7]. Each solution shows the code in R and in Python. Where R code without the tidyverse would differ, an R (base) tab follows the R tab, which uses the tidyverse.
ExrExercise 1 (The average hourly count of bike rentals)
Exercise 1 (The average hourly count of bike rentals) The Bikeshare data count the bikes rented in each of many hours, as bikers. A first look at a count outcome starts with its typical size.
TipHint
mean() in R and the pandas .mean() method in Python compute a sample mean.
Exercise 2 (The weather with the most rentals) The Bikeshare data record the weather of each hour as weathersit, and the number of bikes rented in that hour as bikers. A covariate whose groups have very different sample means of the outcome is a promising one for a regression model.
# The sample mean of the hourly count for each weather, largest first.bikers_by_weather<-bikeshare_raw|>dplyr::group_by(weathersit)|>dplyr::summarise(mean_bikers =mean(bikers))|>dplyr::arrange(dplyr::desc(mean_bikers))bikers_by_weather
Show code
# The sample mean of the hourly count for each weather, largest first.bikers_by_weather_base<-tapply(bikeshare_raw$bikers,bikeshare_raw$weathersit,mean)bikers_by_weather_base<-sort(bikers_by_weather_base, decreasing =TRUE)round(bikers_by_weather_base, 1)#> clear cloudy/misty light rain/snow heavy rain/snow #> 155.0 135.5 86.4 36.0
Show code
# The sample mean of the hourly count for each weather, largest first.bikers_by_weather = ( bikeshare_raw.groupby("weathersit")["bikers"].mean().sort_values(ascending=False))bikers_by_weather.round(1)#> weathersit#> clear 155.0#> cloudy/misty 135.5#> light rain/snow 86.4#> heavy rain/snow 36.0#> Name: bikers, dtype: float64
The sample mean of bikers is largest for clear hours, at 155 bikes. It is smallest for heavy rain/snow hours, at 36 bikes, so weathersit is a promising covariate.
Exercise 3 (A histogram of the hourly count) The Bikeshare data count the bikes rented in each of many hours, as bikers. A histogram shows the shape of the whole set of counts.
bikeshare_raw|>ggplot2::ggplot()+ggplot2::aes(x =bikers)+ggplot2::geom_histogram(bins =30)+# check-unexplained-choices: allowggplot2::labs(x ="Bikes rented in the hour", y ="Number of hours")
Show code
# 30 bars of equal width, from the smallest count to the largest.bikers_range<-range(bikeshare_raw$bikers)bikers_breaks<-seq(bikers_range[1],bikers_range[2], length.out =31# check-unexplained-choices: allow)hist(bikeshare_raw$bikers, breaks =bikers_breaks, # check-unexplained-choices: allow main =NULL, xlab ="Bikes rented in the hour", ylab ="Number of hours")
Show code
import matplotlib.pyplot as pltfig, ax = plt.subplots(figsize=(6, 4), layout="constrained")ax.hist(bikeshare_raw["bikers"], bins=30);# check-unexplained-choices: allowax.set_xlabel("Bikes rented in the hour");ax.set_ylabel("Number of hours");plt.show()
I chose 30 bars, which is about 288 hours per bar. Far fewer bars would hide the long right tail inside a few wide bars, and far more would leave most bars with only a few hours, so the outline would look ragged.
Exercise 4 (Rentals against temperature) The Bikeshare data record the scaled temperature of each hour as temp, and the number of bikes rented in that hour as bikers. A scatterplot shows how the counts change with a numerical covariate.
# Small, faint points, because thousands of hours overlap.bikeshare_raw|>ggplot2::ggplot()+ggplot2::aes(x =temp, y =bikers)+ggplot2::geom_point(size =0.5, alpha =0.3)+ggplot2::labs( x ="Temperature (scaled to 0 to 1)", y ="Bikes rented in the hour")
Show code
# Small, faint points, because thousands of hours overlap.faint_black<-grDevices::adjustcolor("black", alpha.f =0.3)plot(bikeshare_raw$temp,bikeshare_raw$bikers, pch =16, cex =0.3, col =faint_black, xlab ="Temperature (scaled to 0 to 1)", ylab ="Bikes rented in the hour")
Show code
# Small, faint points, because thousands of hours overlap.fig, ax = plt.subplots(figsize=(6, 4), layout="constrained")ax.scatter(bikeshare_raw["temp"], bikeshare_raw["bikers"], s=4, alpha=0.3);ax.set_xlabel("Temperature (scaled to 0 to 1)");ax.set_ylabel("Bikes rented in the hour");plt.show()
ExrExercise 5 (Fit a Poisson regression to bike rentals)
Exercise 5 (Fit a Poisson regression to bike rentals) The Bikeshare data count the bikes rented in each of many hours. Each count covers one hour, so every count has the same exposure magnitude. Consider a Poisson regression model of the hourly count bikers on the covariatestemp, workingday, and weathersit, with clear as the reference level of weathersit.
TipHint
In R, glm() with family = poissonfits the model. In Python, the statsmodels function smf.glm() fits it with family=sm.families.Poisson(). Both code weathersit as one dummy variable for each level except the reference level.
# Make `clear` the first level of weathersit,# so that glm() uses it as the reference level.weather_levels<-c("clear", "cloudy/misty","light rain/snow", "heavy rain/snow")bike<-readr::read_csv("data/bikeshare.csv", show_col_types =FALSE)|>dplyr::mutate(weathersit =factor(weathersit, levels =weather_levels))# Poisson regression of the hourly count on the three covariates.pois_r<-glm(bikers~temp+workingday+weathersit, family =poisson, data =bike)pois_coef<-coef(pois_r)pois_r|>parameters::parameters()|>parameters::print_md()
Parameter
Log-Mean
SE
95% CI
z
p
(Intercept)
3.89
3.23e-03
(3.88, 3.89)
1202.49
< .001
temp
2.13
4.79e-03
(2.12, 2.14)
444.59
< .001
workingday
-8.89e-03
1.94e-03
(-0.01, -5.08e-03)
-4.57
< .001
weathersit (cloudy/misty)
-0.04
2.13e-03
(-0.05, -0.04)
-19.81
< .001
weathersit (light rain/snow)
-0.43
4.02e-03
(-0.44, -0.42)
-107.41
< .001
weathersit (heavy rain/snow)
-0.76
0.17
(-1.11, -0.45)
-4.57
< .001
Show code
# Make `clear` the first level of weathersit,# so that glm() uses it as the reference level.weather_levels<-c("clear", "cloudy/misty","light rain/snow", "heavy rain/snow")# nolint next: undesirable_function_linter. This tab shows base R.bike_base<-read.csv("data/bikeshare.csv")bike_base$weathersit<-factor(bike_base$weathersit, levels =weather_levels)# Poisson regression of the hourly count on the three covariates.pois_base<-glm(bikers~temp+workingday+weathersit, family =poisson, data =bike_base)pois_base_summary<-summary(pois_base)round(pois_base_summary$coefficients, 3)#> Estimate Std. Error z value Pr(>|z|)#> (Intercept) 3.885 0.003 1202.490 0#> temp 2.129 0.005 444.587 0#> workingday -0.009 0.002 -4.574 0#> weathersitcloudy/misty -0.042 0.002 -19.805 0#> weathersitlight rain/snow -0.432 0.004 -107.412 0#> weathersitheavy rain/snow -0.761 0.167 -4.566 0
Show code
import pandas as pdimport statsmodels.api as smimport statsmodels.formula.api as smfbike = pd.read_csv("data/bikeshare.csv")# Poisson regression of the hourly count on the three covariates,# with `clear` as the reference level of weathersit.pois_py = smf.glm("bikers ~ temp + workingday + C(weathersit, Treatment('clear'))", data=bike, family=sm.families.Poisson(),).fit()pois_py.params.round(3)#> Intercept 3.885#> C(weathersit, Treatment('clear'))[T.cloudy/misty] -0.042#> C(weathersit, Treatment('clear'))[T.heavy rain/snow] -0.761#> C(weathersit, Treatment('clear'))[T.light rain/snow] -0.432#> temp 2.129#> workingday -0.009#> dtype: float64
All three tabs fit the same model with the same reference level, clear, so their coefficient estimates agree. The coefficient estimate of temp is 2.13, a positive number, so the event rate of the fitted model rises with the temperature.
ExrExercise 6 (Interpret the temperature coefficient)
Exercise 6 (Interpret the temperature coefficient) Use the Poisson regression model of Exercise 5. Consider two hours with the same workingday and weathersit, one with temp\(0.1\) higher than the other.
# The rate ratio for a 0.1 rise in temp is exp(0.1 * coefficient of temp).ratio_r<-exp(0.1*pois_coef[["temp"]])round(ratio_r, 3)#> [1] 1.237
Show code
import numpy as np# The rate ratio for a 0.1 rise in temp is exp(0.1 * coefficient of temp).ratio_py =float(np.exp(0.1* pois_py.params["temp"]))round(ratio_py, 3)#> 1.237
By Theorem 1 with \(a - b = 0.1\), the rate ratio is \({\operatorname{exp}\mathopen{}\left\{0.1 \beta_{\text{temp}}\right\}\mathclose{}}\), and its estimate is \({\operatorname{exp}\mathopen{}\left\{0.1 \hat{\beta}_{\text{temp}}\right\}\mathclose{}}\). That estimate is 1.24 in the R tab and 1.24 in the Python tab, so the estimatedevent rate of the warmer hour is about 1.24 times that of the cooler hour.
lm() in R and smf.ols() in Python fit the linear regression. fitted() in R returns the fitted values of a model, and the fittedvalues attribute of a statsmodels fit holds them in Python.
# `bike`, `pois_r` and `pois_py` come from the solution to the fit exercise.# Linear regression on the same covariates as the Poisson model.lin_r<-lm(bikers~temp+workingday+weathersit, data =bike)# Count the hours whose fitted value is below zero, for each model.fitted_lin_r<-fitted(lin_r)fitted_pois_r<-fitted(pois_r)n_neg_lin_r<-sum(fitted_lin_r<0)n_neg_pois_r<-sum(fitted_pois_r<0)c(linear =n_neg_lin_r, poisson =n_neg_pois_r)#> linear poisson #> 20 0
Show code
# `bike`, `pois_r` and `pois_py` come from the solution to the fit exercise.# Linear regression on the same covariates as the Poisson model.lin_py = smf.ols("bikers ~ temp + workingday + C(weathersit, Treatment('clear'))", data=bike,).fit()# Count the hours whose fitted value is below zero, for each model.n_neg_lin_py =int((lin_py.fittedvalues <0).sum())n_neg_pois_py =int((pois_py.fittedvalues <0).sum()){"linear": n_neg_lin_py, "poisson": n_neg_pois_py}#> {'linear': 20, 'poisson': 0}
G. James, D. Witten, T. Hastie, and R. Tibshirani, An introduction to statistical learning: With applications in R, 2nd ed. Springer, 2021. doi: 10.1007/978-1-0716-1418-1.
---title: "Poisson Regression"format: html: default revealjs: output-file: poisson-regression-slides.html shift-heading-level-by: 1 docx: output-file: poisson-regression-handout.docxresources: - data/bikeshare.csv---{{< include sds/_subfiles/shared-config.qmd >}}```{r}#| label: python-packages#| include: falsepy_packages <-c("matplotlib", "numpy", "pandas", "statsmodels")reticulate::py_require(py_packages)```# The modelA count [outcome](model-validation.qmd#def-outcome-vector),such as the number of bike rentals in an hour,is not well described by a [normal distribution](https://morrison-lab.github.io/pds/random-variables.html#def-normal)when the counts are small,since the normal distribution puts [probability](https://morrison-lab.github.io/pds/probability-basics.html#def-probability) below zero,and a count is never below zero.A [Poisson regression model](poisson-regression.qmd#def-poisson-regression) takes the [event rate](https://morrison-lab.github.io/pds/distributions.html#def-event-rate)to be the [exponential function](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-exponential-function) of a [linear function](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-linear-function) of the [covariates](model-validation.qmd#def-covariate).{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_def-poisson-regression.qmd >}}# Rate ratios{{< include sds/_subfiles/poisson-regression/_def-rate-ratio.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_thm-poisson-rate-ratio.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_exm-poisson-rate-ratio.qmd >}}# Lab: Poisson regression on bike rentals {#sec-bikeshare-lab}```{r}#| include: falsebikeshare_raw <- readr::read_csv("data/bikeshare.csv", show_col_types =FALSE)bikeshare_n <-nrow(bikeshare_raw)```Hourly counts of bikes rented in a bike-sharing program in Washington, D.C.,on `r bikeshare_n` hours make up the `Bikeshare` data [@james2021islr2e, Section 4.6.2].The data are in [`data/bikeshare.csv`](data/bikeshare.csv), one line per hour,copied from the book's `ISLR2` R package.The exercises use these variables:- `bikers`: the [outcome](model-validation.qmd#def-outcome-vector), the number of bikes rented in that hour.- `temp`: the temperature, scaled to run from $0$ to $1$.- `workingday`: `1` on a working day and `0` otherwise.- `weathersit`: the weather, one of `clear`, `cloudy/misty`, `light rain/snow`, and `heavy rain/snow`.::: {.panel-tabset group="language"}##### R```{r}#| code-fold: show# The first rows of the four columns that the exercises use.bikeshare_raw |> dplyr::select(bikers, temp, workingday, weathersit) |>head()```##### Python```{python}#| code-fold: showimport pandas as pd# The first rows of the four columns that the exercises use.bikeshare_raw = pd.read_csv("data/bikeshare.csv")bikeshare_raw[["bikers", "temp", "workingday", "weathersit"]].head(6)```:::The exercises follow the [Poisson regression](poisson-regression.qmd#def-poisson-regression) part of the book's Chapter 4 lab [@james2021islr2e, Section 4.7.7].Each solution shows the code in R and in Python.Where R code without the tidyverse would differ,an `R (base)` tab follows the `R` tab, which uses the tidyverse.{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_exr-bikeshare-mean.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_sol-bikeshare-mean.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_exr-bikeshare-weather.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_sol-bikeshare-weather.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_exr-bikeshare-histogram.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_sol-bikeshare-histogram.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_exr-bikeshare-temp.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_sol-bikeshare-temp.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_exr-bikeshare-fit.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_sol-bikeshare-fit.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_exr-bikeshare-ratio.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_sol-bikeshare-ratio.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_exr-bikeshare-negative.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/poisson-regression/_sol-bikeshare-negative.qmd >}}