Exponential Families

Published

Last modified: 2026-10-27 23:24:14 (PDT)

Many of the distributions in this course, such as the Bernoulli, Poisson and Normal distributions, share one algebraic form for their probability mass function or density. Distributions with that form are called an exponential family, and they share properties that make estimation easier. This page defines the form and its natural parameter, and checks the form for the Poisson distribution on data.

1 The exponential family form

Definition 1 (Exponential family)

Definition 1 (Exponential family) Let \(X\) be a random variable whose distribution depends on one parameter \(\eta\), and let \(\operatorname{f}\mathopen{}\left(x; \eta\right)\mathclose{}\) be its probability mass function or density. The distributions of \(X\) form an exponential family if there are a nonnegative function \(h\) and a function \(T\) of \(x\) alone, and a function \(A\) of \(\eta\) alone, such that

\[ \operatorname{f}\mathopen{}\left(x; \eta\right)\mathclose{} = {{h\mathopen{}\left(x\right)\mathclose{}}} \, {\operatorname{exp}\mathopen{}\left\{\eta\, {T\mathopen{}\left(x\right)\mathclose{}} - {A\mathopen{}\left(\eta\right)\mathclose{}}\right\}\mathclose{}} \tag{1}\]

for every value \(x\) and every allowed \(\eta\).

Remark 1 (Source: Exponential family)

Remark 1 (Source: Exponential family). Casella and Berger [1, Sec. 3.4], Eq. 3.4.1 write the family as a sum of \(k\) terms inside the exponential, each a function of the parameter times a function of \(x\); Dobson and Barnett [2, Sec. 3.2], Eq. 3.3 write it in the form \({\operatorname{exp}\mathopen{}\left\{a(y) b(\theta) + c(\theta) + d(y)\right\}\mathclose{}}\). Equation 1 is the case of one term. It is Dobson and Barnett’s canonical form when \({T\mathopen{}\left(x\right)\mathclose{}} = x\), and the natural parameter \(\eta\) plays the role of their \(b(\theta)\). See the notation choices for how the symbols line up.

Example 1 (The Poisson distribution is an exponential family)

Example 1 (The Poisson distribution is an exponential family) Let \(X\sim \operatorname{Pois}({\lambda})\), the Poisson distribution with rate \({\lambda}> 0\). Its probability mass function has one factor free of \({\lambda}\), and the factors \({\lambda}^{x}\) and \({\operatorname{exp}\mathopen{}\left\{-{\lambda}\right\}\mathclose{}}\). We rewrite it in the form of Equation 1 by writing \({\lambda}^{x}\) as an exponential of a logarithm:

\[ \begin{aligned} \operatorname{f}\mathopen{}\left(x; {\lambda}\right)\mathclose{} &= \frac{{\lambda}^{x} \, {\operatorname{exp}\mathopen{}\left\{-{\lambda}\right\}\mathclose{}}}{x!} && \text{(Poisson mass function)}\\ &= \frac{1}{x!} \, {\operatorname{exp}\mathopen{}\left\{{\operatorname{log}\mathopen{}\left\{{\lambda}^{x}\right\}\mathclose{}}\right\}\mathclose{}} \, {\operatorname{exp}\mathopen{}\left\{-{\lambda}\right\}\mathclose{}} && \text{(a positive number is the exponential of its logarithm)}\\ &= \frac{1}{x!} \, {\operatorname{exp}\mathopen{}\left\{x\, {\operatorname{log}\mathopen{}\left\{{\lambda}\right\}\mathclose{}}\right\}\mathclose{}} \, {\operatorname{exp}\mathopen{}\left\{-{\lambda}\right\}\mathclose{}} && \text{(the logarithm of a power is the exponent times the logarithm)}\\ &= \frac{1}{x!} \, {\operatorname{exp}\mathopen{}\left\{x\, {\operatorname{log}\mathopen{}\left\{{\lambda}\right\}\mathclose{}} - {\lambda}\right\}\mathclose{}} && \text{(the product of exponentials is the exponential of the sum)} \end{aligned} \]

So the Poisson distributions are an exponential family with \({{h\mathopen{}\left(x\right)\mathclose{}}} = 1 / x!\), \({T\mathopen{}\left(x\right)\mathclose{}} = x\), natural parameter \(\eta= {\operatorname{log}\mathopen{}\left\{{\lambda}\right\}\mathclose{}}\), and \({A\mathopen{}\left(\eta\right)\mathclose{}} = {\lambda}= {\operatorname{exp}\mathopen{}\left\{\eta\right\}\mathclose{}}\).

To check the algebra on data, we use the number of tropical cyclones in northeastern Australia in each of 13 seasons [2, Sec. 1.6.5], with the rate set to the sample mean. We evaluate both sides of Equation 1 at every observed count.

R
Python

cyclones <- c(6, 5, 4, 6, 6, 3, 12, 7, 4, 2, 6, 7, 4)
rate <- mean(cyclones)

base_measure <- function(x) 1 / factorial(x)
natural_parameter <- log(rate)
log_partition <- function(eta) exp(eta)

exponential_form <- base_measure(cyclones) *
  exp(natural_parameter * cyclones - log_partition(natural_parameter))

data.frame(
  count = cyclones,
  poisson = dpois(cyclones, rate),
  exponential_form = exponential_form
)

max(abs(dpois(cyclones, rate) - exponential_form))
#> [1] 1.11022e-16

import numpy as np
import pandas as pd
from scipy.special import factorial
from scipy.stats import poisson

cyclones = np.array([6, 5, 4, 6, 6, 3, 12, 7, 4, 2, 6, 7, 4])
rate = cyclones.mean()

natural_parameter = np.log(rate)
exponential_form = (1 / factorial(cyclones)) * np.exp(
    natural_parameter * cyclones - np.exp(natural_parameter)
)

table = pd.DataFrame(
    {
        "count": cyclones,
        "poisson": poisson.pmf(cyclones, rate),
        "exponential_form": exponential_form,
    }
)
print(table)
#>     count   poisson  exponential_form
#> 0       6  0.157645          0.157645
#> 1       5  0.170782          0.170782
#> 2       4  0.154178          0.154178
#> 3       6  0.157645          0.157645
#> 4       6  0.157645          0.157645
#> 5       3  0.111351          0.111351
#> 6      12  0.006839          0.006839
#> 7       7  0.124730          0.124730
#> 8       4  0.154178          0.154178
#> 9       2  0.060315          0.060315
#> 10      6  0.157645          0.157645
#> 11      7  0.124730          0.124730
#> 12      4  0.154178          0.154178
print(np.max(np.abs(poisson.pmf(cyclones, rate) - exponential_form)))
#> 5.551115123125783e-17

The two columns agree to within rounding error: the largest difference is 1.1e-16.

Definition 2 (Natural parameter)

Definition 2 (Natural parameter) In the form of Equation 1 of an exponential family, the parameter \(\eta\) that multiplies the statistic \({T\mathopen{}\left(x\right)\mathclose{}}\) inside the exponential is the natural parameter, also called the canonical parameter [2, Sec. 3.2].

Example 2 (The natural parameter of the Poisson distribution is the log rate)

Example 2 (The natural parameter of the Poisson distribution is the log rate) In Example 1, the Poisson rate \({\lambda}\) and the natural parameter \(\eta\) are related by

\[ \eta= {\operatorname{log}\mathopen{}\left\{{\lambda}\right\}\mathclose{}} \qquad \text{and} \qquad {\lambda}= {\operatorname{exp}\mathopen{}\left\{\eta\right\}\mathclose{}} \]

For the cyclone data, the rate set to the sample mean is 5.538 cyclones per season, so the natural parameter is \(\eta= 1.712\). The natural parameter can be any real number, while the rate must be greater than zero.

2 Further reading

The Fall 2024 course site for UC Berkeley’s Stat 210A, Theoretical Statistics (Will Fithian) has a course reader on exponential families, sufficiency, completeness and unbiased estimation.

References

[1]

G. Casella and R. Berger, Statistical inference, 2nd ed. Cengage Learning, 2002. Available: https://www.cengage.com/c/statistical-inference-2e-casella-berger/9780534243128/

[2]

A. J. Dobson and A. G. Barnett, An introduction to generalized linear models, 4th ed. CRC press, 2018. doi: 10.1201/9781315182780.