Random variables
Definition 1 (Random variable) A random variable is a function from a sample space \(\Omega\) to the real numbers \(\mathbb{R}\) [1]: it assigns a number to each possible outcome of a random experiment.
Remark. We use uppercase letters (\(X\), \(Y\), \(Z\), …) for random variables, and lowercase letters (\(x\), \(y\), \(z\), …) for their realized (observed) values. We write \(\{X = x\}\) for the event \(\{\omega\in \Omega: X(\omega) = x\}\), and similarly for \(\{X \le x\}\) and other conditions on \(X\). A fully rigorous definition also requires the function to be measurable, so that every such set is an event with a probability; that requirement never binds in these notes.
Remark 1 (A random variable as an uncertain quantity). In practice, a random variable is a quantity whose value we cannot deduce precisely from the information available to us at a given time. The number of heads in two coin flips is a random variable before the flips, because nothing we know fixes it. Once we see the flips, we know its value, and for us it is no longer uncertain.
Example 1 (Number of heads in two coin flips)
Flip a fair coin twice. The sample space is \(\Omega= {\mathopen{}\left\{HH, HT, TH, TT\right\}\mathclose{}}\), each outcome with probability \(1/4\). Let \(X_1\) equal \(1\) if the first flip lands heads and \(0\) otherwise, and let \(X_2\) be the same for the second flip. Let \(S= X_1 + X_2\) be the number of heads:
\[ \begin{aligned} S(HH) &= 2, \\ S(HT) &= 1, \\ S(TH) &= 1, \\ S(TT) &= 0 \end{aligned} \]
Then \(\{S= 1\} = {\mathopen{}\left\{HT, TH\right\}\mathclose{}}\), so
\[ \begin{aligned} \Pr(S= 1) &= 1/4 + 1/4 \\ &= 1/2. \end{aligned} \]
Definition 2 (Range of a random variable) The range of a random variable \(X\), denoted \(\mathcal{R}(X)\), is the set of values \(X\) can take:
\[\mathcal{R}(X) \stackrel{\text{def}}{=}{\mathopen{}\left\{X(\omega) : \omega\in \Omega\right\}\mathclose{}}\]
Example 2 (Range of the number of heads) In Example 1, \(\mathcal{R}(X) = {\mathopen{}\left\{0, 1, 2\right\}\mathclose{}}\).
Example 3 (Discrete random variables) The number of heads in Example 1 is discrete, with a finite range. A count with no upper bound, such as the number of emergency-department visits in a day, is discrete with a countably infinite range, \({\mathopen{}\left\{0, 1, 2, \dots\right\}\mathclose{}}\).
Definition 4 (Continuous random variable) A random variable \(X\) is continuous if \(\Pr(X = x) = 0\) for every \(x \in \mathbb{R}\).
Remark. Every continuous random variable in these notes also has a probability density function. Continuous random variables without a density exist, but they do not arise in applications.
Example 4 (A continuous random variable) Let \(X\) satisfy \(\Pr(a \le X \le b) = b - a\) for all \(0 \le a \le b \le 1\). Taking
\[ \begin{aligned} a &= b \\ &= x \end{aligned} \]
gives
\[ \begin{aligned} \Pr(X = x) &= x - x \\ &= 0 \end{aligned} \]
for every \(x \in [0,1]\). Taking \(a = 0\) and \(b = 1\) gives \(\Pr(0 \le X \le 1) = 1\), so by the complement rule \(X\) falls outside \([0, 1]\) with probability 0, and \(\Pr(X = x) = 0\) for every \(x\) outside \([0, 1]\) as well. So \(X\) is continuous. Its range, \([0, 1]\), is uncountable.
Example 5 (A random variable that is neither discrete nor continuous) Some random variables are neither discrete nor continuous: a time to event that equals exactly \(0\) with positive probability, and otherwise spreads over \((0, \infty)\), is one. For instance, let \(U\) be the continuous random variable of Example 4, and let \(T = 1/(1 - U) - 2\) when \(U \in (1/2, 1)\), and \(T = 0\) otherwise. On \({\mathopen{}\left\{U \in (1/2, 1)\right\}\mathclose{}}\), \(T\) is increasing in \(U\) and takes values in \((0, \infty)\), and solving \(t = 1/(1 - u) - 2\) for \(u\) gives \(u = 1 - 1/(t + 2)\). So \({\mathopen{}\left\{T > 0\right\}\mathclose{}} = {\mathopen{}\left\{U \in (1/2, 1)\right\}\mathclose{}}\), and \({\mathopen{}\left\{T = t\right\}\mathclose{}} = {\mathopen{}\left\{U = 1 - 1/(t + 2)\right\}\mathclose{}}\) for each \(t > 0\). Since \(U\) is continuous (Example 4), single points have probability zero:
\[ \begin{aligned} \Pr(U = 1/2) &= \Pr(U = 1) \\ &= 0 \end{aligned} \]
Using these values:
\[ \begin{aligned} \Pr(T > 0) &= \Pr(U \in (1/2, 1)) && \text{(} {\mathopen{}\left\{T > 0\right\}\mathclose{}} = {\mathopen{}\left\{U \in (1/2, 1)\right\}\mathclose{}} \text{)} \\ &= \Pr(1/2 \le U \le 1) - \Pr(U = 1/2) - \Pr(U = 1) && \text{(additivity)} \\ &= \tfrac{1}{2} - 0 - 0 && \text{(} \Pr(a \le U \le b) = b - a \text{)} \\ &= \tfrac{1}{2} && \text{(simplify)} \end{aligned} \]
\(T\) is not continuous, because
\[ \begin{aligned} \Pr(T = 0) &= 1 - \Pr(T > 0) \\ &= 1/2 \end{aligned} \]
by the complement rule.
\(T\) is not discrete either. If it were, its range would be countable, so \({\mathopen{}\left\{T > 0\right\}\mathclose{}}\) would be the disjoint union of the countably many events \({\mathopen{}\left\{T = t\right\}\mathclose{}}\) with \(t \in \mathcal{R}(T)\) and \(t > 0\), and:
\[ \begin{aligned} \Pr(T > 0) &= \sum_{t \in \mathcal{R}(T),\, t > 0} \Pr(T = t) && \text{(countable additivity)} \\ &= \sum_{t \in \mathcal{R}(T),\, t > 0} {\Pr\mathopen{}\left(U = 1 - \tfrac{1}{t + 2}\right)\mathclose{}} && \text{(} {\mathopen{}\left\{T = t\right\}\mathclose{}} = {\mathopen{}\left\{U = 1 - 1/(t + 2)\right\}\mathclose{}} \text{)} \\ &= 0 && \text{(} U \text{ is continuous)} \end{aligned} \]
which contradicts \(\Pr(T > 0) = 1/2\).
Definition 5 (Realization (observed value)) A realization of a random variable \(X\) is the value \(x = X(\omega)\) it takes at the point \(\omega\) of the sample space that the random experiment produced.
Remark. The random variable \(X\) is the rule, before the experiment is run; the realization \(x\) is one number, after it is run. Probabilities such as \(\Pr(X = x)\) describe the rule, and a realization is a fixed number.
Example 6 (Realizations of the number of heads) In Example 1, let \(X\) be the number of heads in two fair coin flips. Run the experiment once and suppose it produces \(\omega= HT\). Then the realization of \(X\) is \(x = X(HT) = 1\). Running the experiment again might produce \(TT\), with realization \(x = 0\). The variable \(X\) is the same both times; only its realization changes.
1 Characteristics of probability distributions
Definition 6 (Probability mass function (PMF)) If \(X\) is a discrete random variable, the probability mass function of \(X\) at value \(x\), denoted \(f(x)\), \(f_X(x)\), \(\mathbb{P}(x)\), \(\mathbb{P}_X(x)\), or \(\mathbb{P}(X=x)\), is the probability that \(X\) takes exactly the value \(x\) [2]:
\[\mathbb{P}(X=x) \stackrel{\text{def}}{=}\Pr(\{X=x\})\]
Example 7 (PMF of the number of heads) For the number of heads \(X\) in two fair coin flips (Example 1):
\[\mathbb{P}(X = 0) = \tfrac{1}{4}, \quad \mathbb{P}(X = 1) = \tfrac{1}{2}, \quad \mathbb{P}(X = 2) = \tfrac{1}{4}\]
Definition 7 (Bernoulli distribution)
A discrete random variable \(X\) has the Bernoulli distribution with parameter \(\pi\in [0, 1]\), written \(X \sim \operatorname{Ber}(\pi)\) [3], if:
\[ \begin{aligned} \Pr(X=x) &\stackrel{\text{def}}{=}\text{1}_{x\in {\mathopen{}\left\{0,1\right\}\mathclose{}}}\pi^x(1-\pi)^{1-x}\\ &= \begin{cases} \pi, & x=1\\ 1-\pi, & x=0 \end{cases} \end{aligned} \]
Example 8 (The first of two coin flips) In Example 1, let \(X_1\) indicate heads on the first flip, so
\[ \begin{aligned} X_1(HH) &= X_1(HT) \\ &= 1 \end{aligned} \]
and
\[ \begin{aligned} X_1(TH) &= X_1(TT) \\ &= 0. \end{aligned} \]
Then
\[ \begin{aligned} \mathbb{P}(X_1 = 1) &= \Pr({\mathopen{}\left\{HH, HT\right\}\mathclose{}}) \\ &= 1/2 \end{aligned} \]
and
\[ \begin{aligned} \mathbb{P}(X_1 = 0) &= \Pr({\mathopen{}\left\{TH, TT\right\}\mathclose{}}) \\ &= 1/2, \end{aligned} \]
so \(X_1 \sim \operatorname{Ber}(1/2)\).
Definition 8 (Probability density function (PDF)) If \(X\) is a continuous random variable, a probability density function of \(X\), denoted \(f(x)\), \(f_X(x)\), \(\mathbb{p}(x)\), \(\mathbb{p}_X(x)\), or \(\mathbb{p}(X=x)\), is a function \(f\) that satisfies [4]:
- \(f(x) \ge 0\) for every \(x\).
- The integral of \(f\) over any interval is the probability that \(X\) falls in that interval: \[\Pr(a \le X \le b) = \int_a^b f(x)\,dx \quad \text{for all } a \le b\]
Definition 9 (Uniform distribution on a set) Let \(S\) be a set with a measure \(\mu\) such that \(\mu(S) \in (0, \infty)\). The uniform distribution on \(S\) is the probability measure
\[\Pr(A) \stackrel{\text{def}}{=}\frac{\mu(A)}{\mu(S)}, \quad A \subseteq S.\]
Definition 10 (Uniform distribution on an interval) The uniform distribution on an interval \([\alpha, \beta]\) (Definition 9 with \(\mu\) the length), with \(\alpha< \beta\), is the distribution of a random variable \(X\), written \(X \sim \text{Uniform}(\alpha, \beta)\), that is continuous with density [5]:
\[ f(x) \stackrel{\text{def}}{=}\begin{cases} \frac{1}{\beta- \alpha}, & \alpha\le x \le \beta\\ 0, & \text{otherwise} \end{cases} \]
Example 9 (Subinterval probabilities of a uniform random variable) For \(X \sim \text{Uniform}(\alpha, \beta)\) (Definition 10) and any subinterval \([a, b] \subseteq [\alpha, \beta]\) (so \(a, b \in [\alpha, \beta]\) and \(a \le b\)):
\[ \begin{aligned} \Pr(a \le X \le b) &= \int_a^b f(x)\,dx && \text{(definition of a probability density function)} \\ &= \int_a^b \frac{1}{\beta- \alpha}\,dx && \text{(} f(x) = \tfrac{1}{\beta- \alpha} \text{ on } [\alpha, \beta] \text{)} \\ &= \frac{b - a}{\beta- \alpha} && \text{(integrate)} \end{aligned} \]
In particular, the probability that \(X\) falls in any subinterval is proportional to that subinterval’s length. For the standard uniform distribution on \([0, 1]\), \(f(x) = 1\) for \(x \in [0, 1]\) and \(f(x) = 0\) otherwise, giving \(\Pr(a \le X \le b) = b - a\) for all \(a, b \in [0, 1]\) with \(a \le b\). An interval reaching outside \([\alpha, \beta]\) adds nothing to the probability, because \(f = 0\) there.
Definition 11 (Density with respect to a measure, Radon–Nikodym derivative)
Let \(\mu\) be a measure on the range of a random variable \(X\), written \(\mathcal{R}(X)\). A density with respect to a measure \(\mu\), or Radon–Nikodym derivative, of the distribution of \(X\) is a non-negative measurable function \(f\) on \(\mathcal{R}(X)\) whose Lebesgue integral against \(\mu\), not its Riemann integral, over each measurable set \(A \subseteq \mathcal{R}(X)\) is the probability that \(X\) falls in \(A\) [6, pp. 213, 423]:
\[\Pr(X \in A) = \int_A f(x)\,d\mu(x)\]
Example 10 (A fair die roll and counting measure)
Let \(X\) be the roll of a fair six-sided die, so \(\mathcal{R}(X) = D = {\mathopen{}\left\{1, 2, 3, 4, 5, 6\right\}\mathclose{}}\), and let \(\mu\) be the counting measure on \(D\). The constant function \(f(x) = \tfrac{1}{6}\) on \(D\) is a density with respect to \(\mu\) of the distribution of \(X\). For each subset \(A \subseteq D\):
\[ \begin{aligned} \int_A f(x)\,d\mu(x) &= \sum_{x \in A} \tfrac{1}{6} && \text{(an integral with respect to counting measure is a sum)} \\ &= \tfrac{\mathopen{}\left|A\right|\mathclose{}}{6} && \text{(} A \text{ has } \mathopen{}\left|A\right|\mathclose{} \text{ terms)} \\ &= \Pr(X \in A) && \text{(the six faces are equally likely)} \end{aligned} \]
The first step is the integral with respect to counting measure. For example, \(A = {\mathopen{}\left\{5, 6\right\}\mathclose{}}\) gives \(\Pr(X \in A) = \tfrac{2}{6} = \tfrac{1}{3}\).
Proposition 1 (A PMF is a density with respect to counting measure) If \(X\) is a discrete random variable with PMF \(f_X\), and \(\mu\) is the counting measure on \(\mathcal{R}(X)\), then \(f_X\) is a density with respect to \(\mu\) of the distribution of \(X\): for every subset \(A \subseteq \mathcal{R}(X)\),
\[\Pr(X \in A) = \int_A f_X(x)\,d\mu(x)\]
Proof. The counting measure is defined on every subset of \(\mathcal{R}(X)\), so every function on \(\mathcal{R}(X)\) is measurable, and \(f_X \ge 0\), because each of its values is a probability. The range \(\mathcal{R}(X)\) is countable, because \(X\) is discrete, so each \(A \subseteq \mathcal{R}(X)\) is countable too, and the event \({\mathopen{}\left\{X \in A\right\}\mathclose{}}\) is the union of the disjoint events \({\mathopen{}\left\{X = x\right\}\mathclose{}}\) for \(x \in A\):
\[ \begin{aligned} \Pr(X \in A) &= \sum_{x \in A} \Pr(X = x) && \text{(countable additivity)} \\ &= \sum_{x \in A} f_X(x) && \text{(definition of the PMF)} \\ &= \int_A f_X(x)\,d\mu(x) && \text{(an integral with respect to counting measure is a sum)} \end{aligned} \]
The first step is countable additivity, and the last is the integral with respect to counting measure.
Example 11 (The number of heads, integrated against counting measure)
For the number of heads \(X\) in two fair coin flips, with the PMF \(f_X\) of Example 7, let \(\mu\) be the counting measure on \(\mathcal{R}(X) = {\mathopen{}\left\{0, 1, 2\right\}\mathclose{}}\). For \(A = {\mathopen{}\left\{1, 2\right\}\mathclose{}}\):
\[ \begin{aligned} \Pr(X \in {\mathopen{}\left\{1, 2\right\}\mathclose{}}) &= \int_{{\mathopen{}\left\{1, 2\right\}\mathclose{}}} f_X(x)\,d\mu(x) && \text{(a PMF is a density with respect to counting measure)} \\ &= f_X(1) + f_X(2) && \text{(an integral with respect to counting measure is a sum)} \\ &= \tfrac{1}{2} + \tfrac{1}{4} && \text{(the PMF of the number of heads)} \\ &= \tfrac{3}{4} && \text{(add)} \end{aligned} \]
The first step is Proposition 1, and the third uses the PMF values in Example 7.
Remark 2 (PMFs and PDFs differ in the reference measure). A PDF plays the role of a PMF with Lebesgue measure in place of counting measure: Definition 8 integrates it with \(dx\). Definition 11 uses the Lebesgue integral, because the Riemann integral has no version against counting measure. Against counting measure, the Lebesgue integral of a non-negative \(f\) over \(A\) adds up the values of \(f\) on \(A\), which is how a PMF gives \(\Pr(X \in A)\). Against Lebesgue measure, the Lebesgue integral of a non-negative \(f\) over \([a, b]\) agrees with its Riemann integral whenever \(f\) has one (mds), so Definition 8 gives the same values with either one. If the CDF \(F\) of a random variable is absolutely continuous, then \(F'\) is a Radon–Nikodym derivative of its distribution against Lebesgue measure [6, p. 423]. So a PMF and a PDF are both densities with respect to a measure, for different reference measures. Because the two share that role, these notes let \(f_X\) denote either one, and the joint-distribution form of Fubini–Tonelli covers PMFs, PDFs, and density-mass functions in one statement.
Theorem 1 (A density is not unique) If \(f\) is a density of a continuous random variable \(X\), and \(g \ge 0\) differs from \(f\) at only finitely many points, then \(g\) is also a density of \(X\).
Proof. For \(a \le b\), the difference \(g - f\) is \(0\) at all but finitely many points, so:
\[ \begin{aligned} \int_a^b g(x)\,dx &= \int_a^b f(x)\,dx + \int_a^b \mathopen{}\left(g(x) - f(x)\right)\mathclose{}\,dx && \text{(linearity of the integral)} \\ &= \int_a^b f(x)\,dx + 0 && \text{(a function that is 0 at all but finitely many points integrates to 0)} \\ &= \Pr(a \le X \le b) && \text{(} f \text{ is a density of } X \text{)} \end{aligned} \]
Together with \(g \ge 0\), this is Definition 8 for \(g\).
Remark. So the value of a density at a single point carries no probability by itself. These notes use the version that is continuous wherever possible.
Theorem 2 (Some continuous random variables have no density) There exist continuous random variables that have no probability density function.
Remark. The proof needs measure theory beyond these notes. The standard example is the Cantor distribution, which puts no probability on any single point, but puts all of its probability on a set of total length 0, so any candidate density would integrate to 0 over a set of probability 1.
Every continuous random variable in these notes also has a probability density function. Continuous random variables without a density do not arise in applications.
Definition 12 (Normal distribution (Gaussian distribution)) A random variable \(X\) has the normal distribution (or Gaussian distribution) with mean parameter \(\mu\in \mathbb{R}\) and variance parameter \(\sigma^2> 0\), written \(X \sim {\operatorname{N}\mathopen{}\left(\mu, \sigma^2\right)\mathclose{}}\) [7], if \(X\) is continuous with density:
\[\mathbb{p}(X=x) \stackrel{\text{def}}{=}\frac{1}{\sigma\sqrt{2\pi}} \text{e}^{-\frac{(x-\mu)^2}{2\sigma^2}}, \quad x \in \mathbb{R}\]
Example 12 (Normal densities at their centers) For \(X \sim {\operatorname{N}\mathopen{}\left(0, 1\right)\mathclose{}}\), the density at \(x = 0\) is:
\[ \begin{aligned} \mathbb{p}(X=0) &= \frac{1}{1 \cdot\sqrt{2\pi}} \text{e}^{-\frac{(0-0)^2}{2 \cdot 1}} && \text{(normal density with } \mu= 0, \sigma^2= 1 \text{)} \\ &= \frac{1}{\sqrt{2\pi}} && \text{(} \text{e}^0 = 1 \text{)} \\ &\approx 0.399 && \text{(evaluate)} \end{aligned} \]
For \(X \sim {\operatorname{N}\mathopen{}\left(0, 0.1^2\right)\mathclose{}}\), the same steps give \(\mathbb{p}(X=0) = 1 / (0.1\sqrt{2\pi}) \approx 3.99\), a density greater than 1.
Remark. A density is not a probability: it can exceed 1, as the second density in Example 12 does.
Theorem 3 (The normal density integrates to 1) If \(X \sim {\operatorname{N}\mathopen{}\left(\mu, \sigma^2\right)\mathclose{}}\) (Definition 12), then:
\[\int_{-\infty}^{\infty} \frac{1}{\sigma\sqrt{2\pi}} \text{e}^{-\frac{(x-\mu)^2}{2\sigma^2}}\,dx = 1\]
Theorem 4 (The mean and variance of the normal distribution) If \(X \sim {\operatorname{N}\mathopen{}\left(\mu, \sigma^2\right)\mathclose{}}\) (Definition 12), then \(X\) has mean \({\mathbb{E}\mathopen{}\left[X\right]\mathclose{}} = \mu\) and variance \({\operatorname{Var}\mathopen{}\left(X\right)\mathclose{}} = \sigma^2\).
Remark. The proofs of these two theorems evaluate three integrals by calculus: the total of the density, the mean integral and the variance integral. The total is the Gaussian integral. This page omits the proofs [8]. These facts are where the parameters’ names come from.
Definition 13 (Standard normal distribution) The standard normal distribution is the normal distribution with mean parameter 0 and variance parameter 1, \({\operatorname{N}\mathopen{}\left(0, 1\right)\mathclose{}}\).
Example 13 (Half of a standard normal lies below 0) If \(Z\) has the standard normal distribution, its density \(\mathbb{p}(Z=z) = \frac{1}{\sqrt{2\pi}} \text{e}^{-z^2/2}\) takes the same value at \(z\) and \(-z\), so \(\Pr(Z \le 0) = \Pr(Z \ge 0)\). Those two probabilities sum to 1, because \(\Pr(Z = 0) = 0\) for a continuous \(Z\), so each is \(\tfrac{1}{2}\).
Example 14 (Probability of an event within one time unit) Let \(T\) be exponential with rate \({\lambda}\). The probability that \(T\) falls in \([0, 1]\) is:
\[ \begin{aligned} \Pr(0 \le T \le 1) &= \int_0^1 {\lambda}\text{e}^{-{\lambda}t}\,dt && \text{(definition of a density)} \\ &= \mathopen{}\left[-\text{e}^{-{\lambda}t}\right]\mathclose{}_{t=0}^{1} && \text{(antiderivative of } {\lambda}\text{e}^{-{\lambda}t} \text{)} \\ &= 1 - \text{e}^{-{\lambda}} && \text{(evaluate at the bounds)} \end{aligned} \]
With \({\lambda}= 0.5\), for example, \(\Pr(0 \le T \le 1) = 1 - \text{e}^{-0.5} \approx 0.393\).
Definition 15 (Cumulative distribution function (CDF)) For a random variable \(X\) (discrete or continuous), its cumulative distribution function [10] is:
\[{\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} \stackrel{\text{def}}{=}\Pr(X\le t), \quad t\in\mathbb{R}.\]
Remark. The CDF is always defined, whether \(X\) is discrete or continuous.
Example 15 (CDF of the number of heads) Summing the PMF in Example 7 over values at or below \(t\):
\[ {\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} = \begin{cases} 0, & t < 0 \\ \tfrac{1}{4}, & 0 \le t < 1 \\ \tfrac{3}{4}, & 1 \le t < 2 \\ 1, & t \ge 2 \end{cases} \]
Theorem 5 (Properties of a CDF) The CDF \(\mathbb{F}\) of any random variable \(X\) is non-decreasing, and:
\[\lim_{t \to -\infty} {\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} = 0, \qquad \lim_{t \to \infty} {\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} = 1\]
Proof. Non-decreasing. For \(s < t\), \(\{X \le t\}\) is the disjoint union of \(\{X \le s\}\) and \(\{s < X \le t\}\), so finite additivity gives:
\[ \begin{aligned} {\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} &= \Pr(X \le s) + \Pr(s < X \le t) && \text{(additivity, and the definition of the CDF)} \\ &\ge \Pr(X \le s) && \text{(probabilities are non-negative)} \\ &= {\mathbb{F}\mathopen{}\left(s\right)\mathclose{}} && \text{(definition of the CDF)} \end{aligned} \]
Limit at \(-\infty\). The events \(A_n = \{X \le -n\}\), \(n = 1, 2, \ldots\), are decreasing, and their intersection is empty, because every outcome \(\omega\) has \(X(\omega) > -n\) once \(n > -X(\omega)\). By continuity of probability, \({\mathbb{F}\mathopen{}\left(-n\right)\mathclose{}} = \Pr(A_n) \to 0\). For \(t \le -n\), \(0 \le {\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} \le {\mathbb{F}\mathopen{}\left(-n\right)\mathclose{}}\), because \(\mathbb{F}\) is non-decreasing, so \({\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} \to 0\) as \(t \to -\infty\).
Limit at \(\infty\). The events \(B_n = \{X > n\}\) are decreasing, and their intersection is empty, because every outcome \(\omega\) has \(X(\omega) \le n\) once \(n \ge X(\omega)\). By continuity of probability, \(\Pr(B_n) \to 0\), so, by the complement rule:
\[ \begin{aligned} {\mathbb{F}\mathopen{}\left(n\right)\mathclose{}} &= 1 - \Pr(X > n) && \text{(complement rule)} \\ &\to 1 - 0 && \text{(} \Pr(B_n) \to 0 \text{)} \end{aligned} \]
For \(t \ge n\), \({\mathbb{F}\mathopen{}\left(n\right)\mathclose{}} \le {\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} \le 1\), because \(\mathbb{F}\) is non-decreasing, so \({\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} \to 1\) as \(t \to \infty\).
Theorem 6 (The CDF determines the distribution) If random variables \(X\) and \(Y\) have the same CDF, then \(\Pr(X \in B) = \Pr(Y \in B)\) for every Borel set \(B \subseteq \mathbb{R}\), which includes every interval, and every set built from intervals by countably many unions, intersections, and complements.
Definition 16 (Quantile function (population inverse CDF)) For a random variable \(X\) with cumulative distribution function (CDF) \(\mathbb{F}\), its population quantile function (generalized inverse of \(\mathbb{F}\)) is [11]:
\[{\mathbb{Q}\mathopen{}\left(p\right)\mathclose{}} \stackrel{\text{def}}{=}\inf {\mathopen{}\left\{t : {\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} \ge p\right\}\mathclose{}}, \quad p \in (0, 1).\]
Remark. The restriction to \(p \in (0, 1)\) keeps \({\mathbb{Q}\mathopen{}\left(p\right)\mathclose{}}\) finite: for \(p = 1\), the set \({\mathopen{}\left\{t : {\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} \ge 1\right\}\mathclose{}}\) is empty whenever \({\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} < 1\) for every \(t\), as for a normal distribution, and the infimum of the empty set is \(+\infty\).
Example 16 (Median of the number of heads) From Example 15, \({\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} \ge 0.5\) first holds at \(t = 1\), so the median of the number of heads is \({\mathbb{Q}\mathopen{}\left(0.5\right)\mathclose{}} = 1\).
Remark. Figure 1 shows the mass function, CDF and quantile function of the number of heads in two fair coin flips (top row), beside the density, CDF and quantile function of a standard normal variable (bottom row).
Theorem 7 (A density integrates to the CDF) If \(X\) is a continuous random variable with density \(f\) and CDF \(\mathbb{F}\), then for every \(t\):
\[{\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} = \int_{-\infty}^{t} f(x)\,dx\]
Proof. For any \(a < t\), \(\{X \le t\}\) is the disjoint union of \(\{X \le a\}\) and \(\{a < X \le t\}\), and \(\Pr(X = a) = 0\) because \(X\) is continuous. So:
\[ \begin{aligned} {\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} &= \Pr(X \le a) + \Pr(a < X \le t) && \text{(additivity, and the definition of the CDF)} \\ &= {\mathbb{F}\mathopen{}\left(a\right)\mathclose{}} + \Pr(a \le X \le t) && \text{(definition of the CDF; } \Pr(X = a) = 0 \text{)} \\ &= {\mathbb{F}\mathopen{}\left(a\right)\mathclose{}} + \int_a^t f(x)\,dx && \text{(definition of the density)} \end{aligned} \]
Letting \(a \to -\infty\), \({\mathbb{F}\mathopen{}\left(a\right)\mathclose{}} \to 0\) (Theorem 5), which gives \({\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} = \int_{-\infty}^{t} f(x)\,dx\).
Theorem 8 (The derivative of the CDF is the density where the density is continuous) If \(X\) is a continuous random variable with density \(f\) and CDF \(\mathbb{F}\), then at every \(t\) where \(f\) is continuous:
\[f(t) = \frac{\partial}{\partial t} {\mathbb{F}\mathopen{}\left(t\right)\mathclose{}}\]
Proof. By Theorem 7, \({\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} = \int_{-\infty}^{t} f(x)\,dx\). The fundamental theorem of calculus then gives \(\frac{\partial}{\partial t} \int_{-\infty}^{t} f(x)\,dx = f(t)\) at every \(t\) where \(f\) is continuous.
Corollary 1 (A CDF has its density as derivative at all but finitely many points) If \(X\) is a continuous random variable with CDF \(\mathbb{F}\) and a density \(f\) that is continuous at all but finitely many points, then \(\mathbb{F}\) is differentiable with \(\mathbb{F}' = f\) at all but finitely many points.
Proof. At every point \(t\) where \(f\) is continuous, Theorem 8 gives \(\mathbb{F}'(t) = f(t)\), and by assumption those are all but finitely many points.
Remark. Every density in these notes is continuous at all but finitely many points, so Corollary 1 applies to each of them.
Example 17 (CDF and density of a uniform random variable) For \(X \sim \text{Uniform}(0, 1)\) with the density \(f\) of Definition 10, and \(t \in [0, 1]\):
\[ \begin{aligned} {\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} &= \int_{-\infty}^{t} f(x)\,dx && \text{(the CDF is the integral of the density)} \\ &= \int_{-\infty}^{0} 0\,dx + \int_{0}^{t} 1\,dx && \text{(split at 0, and substitute } f \text{)} \\ &= t && \text{(integrate)} \end{aligned} \]
For \(t \in (0, 1)\), where \(f\) is continuous,
\[ \begin{aligned} \frac{\partial}{\partial t} {\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} &= 1 \\ &= f(t). \end{aligned} \]
At \(t = 1\), \(f\) jumps from 1 to 0, and \(\mathbb{F}\) has no derivative: its slope is 1 on the left and 0 on the right.
Theorem 9 (A piecewise-smooth CDF has its derivative as a density) If the CDF \(\mathbb{F}\) of a random variable \(X\) is continuous everywhere, and has a continuous derivative at all but finitely many points, then \(X\) is continuous, and \(\mathbb{F}'\) (given any values at the finitely many exceptional points) is a density of \(X\).
Proof. For events \(A \subseteq B\), \(B\) is the disjoint union of \(A\) and \(B \setminus A\), so additivity gives \(\Pr(A) \le \Pr(B)\). For any \(x\) and \(\Delta> 0\), \(\{X = x\} \subseteq \{x - \Delta< X \le x\}\), so:
\[ \begin{aligned} 0 &\le \Pr(X = x) && \text{(probabilities are non-negative)} \\ &\le \Pr(x - \Delta< X \le x) && \text{(} \{X = x\} \subseteq \{x - \Delta< X \le x\} \text{)} \\ &= {\mathbb{F}\mathopen{}\left(x\right)\mathclose{}} - {\mathbb{F}\mathopen{}\left(x - \Delta\right)\mathclose{}} && \text{(additivity, and the definition of the CDF)} \end{aligned} \]
and \({\mathbb{F}\mathopen{}\left(x\right)\mathclose{}} - {\mathbb{F}\mathopen{}\left(x - \Delta\right)\mathclose{}} \to 0\) as \(\Delta\downarrow 0\), because \(\mathbb{F}\) is continuous. So \(\Pr(X = x) = 0\) for every \(x\), and \(X\) is continuous. Then, for \(a \le b\):
\[ \begin{aligned} \Pr(a \le X \le b) &= \Pr(a < X \le b) && \text{(} \Pr(X = a) = 0 \text{)} \\ &= {\mathbb{F}\mathopen{}\left(b\right)\mathclose{}} - {\mathbb{F}\mathopen{}\left(a\right)\mathclose{}} && \text{(additivity, and the definition of the CDF)} \\ &= \int_a^b \mathbb{F}'(x)\,dx && \text{(fundamental theorem of calculus)} \end{aligned} \]
The last step applies the fundamental theorem of calculus on each piece of \([a, b]\) between exceptional points, where \(\mathbb{F}'\) is continuous, and joins the pieces using the continuity of \(\mathbb{F}\) at the exceptional points. Finally, \(\mathbb{F}' \ge 0\) wherever it exists, because \(\mathbb{F}\) is non-decreasing (Theorem 5).
Example 18 (A density from a CDF with a corner) Let \(X\) have CDF \({\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} = 0\) for \(t < 0\), \({\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} = t^2\) for \(0 \le t \le 1\), and \({\mathbb{F}\mathopen{}\left(t\right)\mathclose{}} = 1\) for \(t > 1\). This \(\mathbb{F}\) is continuous everywhere, and has a continuous derivative everywhere except \(t = 1\), where its slope drops from 2 to 0. By Theorem 9, \(X\) is continuous with density \(f(t) = 2t\) for \(0 \le t < 1\) and \(f(t) = 0\) otherwise. As a check, \(\int_0^1 2t\,dt = 1\).
Theorem 10 (A density as a limit of interval probabilities) If \(X\) is a continuous random variable with density \(f\), and \(f\) is continuous at \(x\), then \(f(x)\) is the limit of the probability that \(X\) falls in an interval starting at \(x\), divided by the width of that interval, as that width shrinks to 0:
\[f(x) = \lim_{\Delta\downarrow 0} \frac{\Pr(x \le X < x + \Delta)}{\Delta}\]
Proof. For \(\Delta> 0\):
\[ \begin{aligned} \frac{\Pr(x \le X < x + \Delta)}{\Delta} &= \frac{\Pr(x < X \le x + \Delta)}{\Delta} && \text{(} \Pr(X = x) = 0 \text{ and } \Pr(X = x + \Delta) = 0 \text{)} \\ &= \frac{{\mathbb{F}\mathopen{}\left(x + \Delta\right)\mathclose{}} - {\mathbb{F}\mathopen{}\left(x\right)\mathclose{}}}{\Delta} && \text{(additivity, and the definition of the CDF)} \end{aligned} \]
By Theorem 8, \(\mathbb{F}\) is differentiable at \(x\) with \(\mathbb{F}'(x) = f(x)\), so this difference quotient converges to \(f(x)\) as \(\Delta\downarrow 0\).
Example 19 (The uniform density as a limit) For \(X \sim \text{Uniform}(0, 1)\) and \(x \in [0, 1)\), \(\Pr(x \le X < x + \Delta) = \Delta\) once \(\Delta\le 1 - x\), so:
\[ \begin{aligned} f(x) &= \lim_{\Delta\downarrow 0} \frac{\Delta}{\Delta} \\ &= 1 \end{aligned} \]
For \(x < 0\) or \(x > 1\), the interval eventually misses \([0, 1]\), so the limit is \(0\). Both agree with the density of Definition 10.
Proposition 2 (The density limit is the right derivative of the CDF) If \(X\) is a continuous random variable with CDF \(\mathbb{F}\), then for every \(x\):
\[\lim_{\Delta\downarrow 0} \frac{\Pr(x \le X < x + \Delta)}{\Delta} = \lim_{\Delta\downarrow 0} \frac{{\mathbb{F}\mathopen{}\left(x + \Delta\right)\mathclose{}} - {\mathbb{F}\mathopen{}\left(x\right)\mathclose{}}}{\Delta}\]
whenever either limit exists; the right-hand side is the right derivative of \(\mathbb{F}\) at \(x\).
Proof. The first two lines of the proof of Theorem 10 use only that \(X\) is continuous, so for every \(\Delta> 0\) the two quotients are equal. Equal functions of \(\Delta\) have the same limit as \(\Delta\downarrow 0\), or both have none.
Example 20 (The density limit at a jump) At a point where \(f\) is not continuous, the limit can differ from the chosen value \(f(x)\). For \(X \sim \text{Uniform}(0, 1)\) with the density \(f\) of Definition 10, which sets \(f(1) = 1\) and \(f(x) = 0\) for \(x > 1\), and for \(\Delta> 0\):
\[ \begin{aligned} \Pr(1 \le X < 1 + \Delta) &= \Pr(1 \le X \le 1 + \Delta) && \text{(} \Pr(X = 1 + \Delta) = 0 \text{)} \\ &= \int_1^{1 + \Delta} f(x)\,dx && \text{(definition of the density)} \\ &= 0 && \text{(} f = 0 \text{ on } (1, 1 + \Delta] \text{; one point does not change an integral)} \end{aligned} \]
So the limit of \(\Pr(1 \le X < 1 + \Delta) / \Delta\) at \(x = 1\) is 0, although \(f(1) = 1\). By Proposition 2, 0 is also the right derivative of \(\mathbb{F}\) at 1, matching the slope of 0 on the right in Example 17.
Theorem 11 (Density functions integrate to 1) If \(X\) is a continuous random variable with density \(f\):
\[\int_{-\infty}^{\infty} f(x)\, dx = 1\]
Proof. \[ \begin{aligned} \int_{-\infty}^{\infty} f(x)\, dx &= \lim_{b \to \infty} \int_{-\infty}^{b} f(x)\, dx && \text{(definition of an improper integral)} \\ &= \lim_{b \to \infty} {\mathbb{F}\mathopen{}\left(b\right)\mathclose{}} && \text{(the CDF is the integral of the density)} \\ &= 1 && \text{(limit of a CDF)} \end{aligned} \]
The last step is Theorem 5.
Example 21 (The uniform density integrates to 1) For \(X \sim \text{Uniform}(0, 1)\) (Definition 10),
\[ \begin{aligned} \int_{-\infty}^{\infty} f(x)\,dx &= \int_0^1 1\,dx \\ &= 1. \end{aligned} \]
Definition 17 (Jointly distributed random variables) Random variables \(X_1, \ldots, X_n\) are jointly distributed when they are defined on the same sample space \(\Omega\), so that an event about several of them at once, such as \(\{X_1 \le x_1\} \cap \{X_2 \le x_2\}\), has a probability.
Remark. We write \(\Pr(X \in A, Y \in B)\) for \(\Pr(\{X \in A\} \cap \{Y \in B\})\), and similarly for more variables.
Example 22 (The first flip and the total) In Example 1, let \(X_1\) indicate heads on the first flip (as in Example 8), and let \(X\) be the total number of heads. Both are functions on the same sample space, \({\mathopen{}\left\{HH, HT, TH, TT\right\}\mathclose{}}\), so they are jointly distributed, and, for example:
\[ \begin{aligned} \Pr(X_1 = 1, X = 1) &= \Pr({\mathopen{}\left\{HH, HT\right\}\mathclose{}} \cap {\mathopen{}\left\{HT, TH\right\}\mathclose{}}) \\ &= \Pr({\mathopen{}\left\{HT\right\}\mathclose{}}) \\ &= \tfrac{1}{4} \end{aligned} \]
Definition 18 (Joint distribution) The joint distribution of jointly distributed random variables \(X_1, \ldots, X_n\) is the function that assigns to each set \(A \subseteq \mathbb{R}^n\) for which \(\{(X_1, \ldots, X_n) \in A\}\) is an event the probability \(\Pr((X_1, \ldots, X_n) \in A)\) [13].
Example 23 (Joint distribution of the first flip and the total) In Example 22, the joint distribution of \(X_1\) and \(X\) assigns to the set \(A = {\mathopen{}\left\{(x_1, x) : x_1 = 1,\, x \ge 1\right\}\mathclose{}}\) the probability
\[ \begin{aligned} \Pr((X_1, X) \in A) &= \Pr({\mathopen{}\left\{HH, HT\right\}\mathclose{}}) \\ &= \tfrac{1}{2} \end{aligned} \]
since the first flip is heads exactly on \(HH\) and \(HT\), and each of those outcomes has at least one head.
Theorem 12 (A joint distribution is determined by its rectangle probabilities) If jointly distributed random variables \(X_1, \ldots, X_n\), and jointly distributed random variables \(Y_1, \ldots, Y_n\), satisfy
\[\Pr(X_1 \in A_1, \ldots, X_n \in A_n) = \Pr(Y_1 \in A_1, \ldots, Y_n \in A_n)\]
for all sets \(A_1, \ldots, A_n\) of real numbers for which these are events, then \((X_1, \ldots, X_n)\) and \((Y_1, \ldots, Y_n)\) have the same joint distribution.
Remark. The proof needs measure theory beyond these notes: the sets \(A_1 \times \cdots \times A_n\) form a \(\pi\)-system, and two probability measures that agree on a \(\pi\)-system agree on the \(\sigma\)-algebra it generates [6].
Definition 19 (Joint probability mass function (joint PMF)) For jointly distributed discrete random variables \(X\) and \(Y\), the joint probability mass function (joint PMF) of \(X\) and \(Y\) is the probability that \(X\) takes the value \(x\) and \(Y\) takes the value \(y\):
\[\mathbb{P}(X = x,\, Y = y) \stackrel{\text{def}}{=}\Pr(\{X = x\} \cap \{Y = y\})\]
Example 24 (Joint PMF of the first flip and the total) Continuing Example 22, each outcome of the two flips fixes both \(X_1\) and \(X\), so the joint PMF puts probability \(1/4\) on each outcome’s pair of values:
| \(X = 0\) | \(X = 1\) | \(X = 2\) | |
|---|---|---|---|
| \(X_1 = 0\) | \(1/4\) (\(TT\)) | \(1/4\) (\(TH\)) | \(0\) |
| \(X_1 = 1\) | \(0\) | \(1/4\) (\(HT\)) | \(1/4\) (\(HH\)) |
Definition 20 (Marginal distribution) When a random variable \(X\) is one of several jointly distributed random variables, the distribution of \(X\) by itself (its CDF, and its PMF or density) is called the marginal distribution of \(X\) [14].
Remark. The word “marginal” only says that the other variables are being set aside; the marginal distribution of \(X\) is the same object as the distribution of \(X\).
Example 25 (Marginal distribution of the total) In Example 24, the marginal distribution of the total \(X\) is the PMF of Example 7: \(\mathbb{P}(X = 0) = 1/4\), \(\mathbb{P}(X = 1) = 1/2\), \(\mathbb{P}(X = 2) = 1/4\).
Theorem 13 (Marginal PMF from a joint PMF) If \(X\) and \(Y\) are jointly distributed discrete random variables, then for every \(x\):
\[\mathbb{P}(X = x) = \sum_{y \in \mathcal{R}(Y)} \mathbb{P}(X = x,\, Y = y)\]
Proof. The event \(\{X = x\}\) is the disjoint union of the events \(\{X = x\} \cap \{Y = y\}\) over the countably many values \(y \in \mathcal{R}(Y)\), so:
\[ \begin{aligned} \mathbb{P}(X = x) &= {\Pr\mathopen{}\left(\bigcup_{y \in \mathcal{R}(Y)} \mathopen{}\left(\{X = x\} \cap \{Y = y\}\right)\mathclose{}\right)\mathclose{}} && \text{(the events partition } \{X = x\} \text{)} \\ &= \sum_{y \in \mathcal{R}(Y)} \Pr(\{X = x\} \cap \{Y = y\}) && \text{(countable additivity)} \\ &= \sum_{y \in \mathcal{R}(Y)} \mathbb{P}(X = x,\, Y = y) && \text{(definition of the joint PMF)} \end{aligned} \]
Example 26 (Summing a row of the joint PMF) In Example 24, summing the \(X_1 = 1\) row:
\[ \begin{aligned} \mathbb{P}(X_1 = 1) &= \mathbb{P}(X_1 = 1, X = 0) + \mathbb{P}(X_1 = 1, X = 1) + \mathbb{P}(X_1 = 1, X = 2) && \text{(marginal PMF from the joint PMF)} \\ &= 0 + \tfrac{1}{4} + \tfrac{1}{4} && \text{(read the table)} \\ &= \tfrac{1}{2} && \text{(add)} \end{aligned} \]
which matches \(X_1 \sim \operatorname{Ber}(1/2)\) from Example 8.
Definition 21 (Joint probability density function (joint density)) For jointly distributed continuous random variables \(X\) and \(Y\), a joint probability density function (joint density) of \(X\) and \(Y\), denoted \(f_{X,Y}(x, y)\) or \(\mathbb{p}(X = x,\, Y = y)\), is a function \(f_{X,Y}\) on \(\mathbb{R}^2\) that satisfies:
- \(f_{X,Y}(x, y) \ge 0\) for every \((x, y)\).
- The integral of \(f_{X,Y}\) over any region \(A \subseteq \mathbb{R}^2\) is the probability that the pair \((X, Y)\) falls in \(A\): \[\Pr((X, Y) \in A) = \iint_A f_{X,Y}(x, y)\,dx\,dy\]
Remark. As with events, a fully rigorous version restricts \(A\) to a designated collection of regions; every region that arises in these notes is in it.
Example 27 (A joint density on a triangle) Let \(f_{X,Y}(x, y) = 2\) for \(0 \le x \le y \le 1\), and \(0\) otherwise. The triangle \(\{(x, y) : 0 \le x \le y \le 1\}\) has area \(1/2\), so \(f_{X,Y}\) gives the whole plane probability \(2 \cdot\tfrac{1}{2} = 1\), and it is the joint density of a pair \((X, Y)\) with \(X \le Y\) always. For example, the probability that both are at most \(1/2\) is \(2\) times the area of the smaller triangle \(\{0 \le x \le y \le 1/2\}\):
\[ \begin{aligned} \Pr(X \le \tfrac{1}{2}, Y \le \tfrac{1}{2}) &= 2 \cdot\tfrac{1}{8} \\ &= \tfrac{1}{4} \end{aligned} \]
Theorem 14 (Not every pair of continuous random variables has a joint density) If \(X\) is a continuous random variable, then the pair \((X, X)\) has no joint density.
Proof. Suppose \(f\) were a joint density of \((X, X)\), and let \(L = {\mathopen{}\left\{(x, y) : y = x\right\}\mathclose{}}\) be the diagonal line. Every outcome \(\omega\) has \(X(\omega) = X(\omega)\), so \({\mathopen{}\left\{(X, X) \in L\right\}\mathclose{}} = \Omega\). For each \(x\), the function \(y \mapsto \text{1}_{y = x} f(x, y)\) is \(0\) except at the single point \(y = x\), so its integral over \(y\) is \(0\). Then:
\[ \begin{aligned} 1 &= \Pr((X, X) \in L) && \text{(} {\mathopen{}\left\{(X, X) \in L\right\}\mathclose{}} = \Omega\text{, and } \Pr(\Omega) = 1 \text{)} \\ &= \iint_L f(x, y)\,dx\,dy && \text{(definition of a joint density)} \\ &= \int_{-\infty}^{\infty} \mathopen{}\left(\int_{-\infty}^{\infty} \text{1}_{y = x} f(x, y)\,dy\right)\mathclose{}\,dx && \text{(iterate the integral; Tonelli's theorem)} \\ &= \int_{-\infty}^{\infty} 0\,dx && \text{(the inner integrand is 0 except at } y = x \text{)} \\ &= 0 && \text{(integrate)} \end{aligned} \]
which is a contradiction. Tonelli’s theorem allows the iterated integral because \(f \ge 0\) (Fubini–Tonelli theorem; Billingsley [6], Theorem 18.3).
Theorem 15 (Marginal density from a joint density) If \(X\) and \(Y\) have joint density \(f_{X,Y}\), then \(X\) has the density:
\[f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x, y)\,dy\]
Proof. For \(a \le b\), the event \(\{a \le X \le b\}\) is the event that \((X, Y)\) falls in the strip \([a, b] \times \mathbb{R}\), so:
\[ \begin{aligned} \Pr(a \le X \le b) &= \Pr((X, Y) \in [a, b] \times \mathbb{R}) && \text{(same event)} \\ &= \iint_{[a, b] \times \mathbb{R}} f_{X,Y}(x, y)\,dx\,dy && \text{(definition of a joint density)} \\ &= \int_a^b \mathopen{}\left(\int_{-\infty}^{\infty} f_{X,Y}(x, y)\,dy\right)\mathclose{}\,dx && \text{(iterate the integral; Tonelli's theorem)} \\ &= \int_a^b f_X(x)\,dx && \text{(definition of } f_X \text{)} \end{aligned} \]
Tonelli’s theorem allows the iterated integral because \(f_{X,Y} \ge 0\) (Fubini–Tonelli theorem; Billingsley [6], Theorem 18.3). So \(f_X\) satisfies Definition 8.
Example 28 (Marginal densities on the triangle) For the joint density of Example 27 and \(x \in [0, 1]\), \(f_{X,Y}(x, y) = 2\) exactly when \(x \le y \le 1\), so:
\[ \begin{aligned} f_X(x) &= \int_{-\infty}^{\infty} f_{X,Y}(x, y)\,dy && \text{(marginal density from a joint density)} \\ &= \int_x^1 2\,dy && \text{(} f_{X,Y}(x, y) = 2 \text{ for } x \le y \le 1 \text{, else } 0 \text{)} \\ &= 2(1 - x) && \text{(integrate)} \end{aligned} \]
The same steps with the roles swapped give
\[ \begin{aligned} f_Y(y) &= \int_0^y 2\,dx \\ &= 2y \end{aligned} \]
for \(y \in [0, 1]\).
Definition 22 (Joint density-mass function) Let \(X\) be a discrete random variable and \(Y\) a continuous random variable, jointly distributed. A joint density-mass function of \(X\) and \(Y\), denoted \(\mathbb{p}(X = x,\, Y = y)\), is a function \(\mathbb{p}(X = x,\, Y = y) \ge 0\) that is a probability mass in \(x\) and a probability density in \(y\): for every \(x\) and every set \(B \subseteq \mathbb{R}\),
\[\Pr(X = x,\, Y \in B) = \int_{B} \mathbb{p}(X = x,\, Y = y)\,dy\]
Remark. When \(X\) is continuous and \(Y\) is discrete, the roles swap: \(\Pr(X \in B,\, Y = y) = \int_B \mathbb{p}(X = x,\, Y = y)\,dx\).
Corollary 2 (Marginal PMF from a joint density-mass function) If \(X\) and \(Y\) have joint density-mass function \(\mathbb{p}(X = x,\, Y = y)\), then for every \(x\):
\[\mathbb{P}(X = x) = \int_{-\infty}^{\infty} \mathbb{p}(X = x,\, Y = y)\,dy\]
Proof. Every outcome has \(Y(\omega) \in \mathbb{R}\), so \({\mathopen{}\left\{Y \in \mathbb{R}\right\}\mathclose{}} = \Omega\), and:
\[ \begin{aligned} \mathbb{P}(X = x) &= \Pr({\mathopen{}\left\{X = x\right\}\mathclose{}} \cap \Omega) && \text{(} {\mathopen{}\left\{X = x\right\}\mathclose{}} \subseteq \Omega\text{)} \\ &= \Pr(X = x,\, Y \in \mathbb{R}) && \text{(} {\mathopen{}\left\{Y \in \mathbb{R}\right\}\mathclose{}} = \Omega\text{)} \\ &= \int_{-\infty}^{\infty} \mathbb{p}(X = x,\, Y = y)\,dy && \text{(definition of a joint density-mass function, with } B = \mathbb{R}\text{)} \end{aligned} \]
Example 29 (A coin flip and a waiting time) Let \(\mathbb{p}(X = 0,\, Y = y) = 1/2\) for \(y \in [0, 1]\), \(\mathbb{p}(X = 1,\, Y = y) = 1/4\) for \(y \in [0, 2]\), and \(0\) otherwise. Integrating over \(y\) gives \(\mathbb{P}(X = 0) = 1/2\) and \(\mathbb{P}(X = 1) = 1/2\), which add to 1. For example, the probability that \(X = 1\) and \(Y \le 1\) is:
\[ \begin{aligned} \Pr(X = 1,\, Y \le 1) &= \int_0^1 \tfrac{1}{4}\,dy \\ &= \tfrac{1}{4} \end{aligned} \]
1.0.1 Joint distributions and marginalization
For a discrete random variable \(X\) taking values in \(\mathbb{X}\), a probability mass function (PMF) \(\mathbb{P}(x) : \mathbb{X} \to [0, 1]\) gives the probability that \(X\) takes the value \(x\):
\[\sum_{x \in \mathbb{X}} \mathbb{P}(x) = 1\]
The support of the distribution is the subset of values where the probability is strictly positive: \(\{x \in \mathbb{X} : \mathbb{P}(x) > 0\}\). When \(\mathbb{X}\) is finite, \(\mathbb{P}(x)\) can be written as a probability vector.
For two discrete variables \(X \in \mathbb{X}\) and \(Y \in \mathbb{Y}\), their joint probability distribution \(\mathbb{P}(x, y) : \mathbb{X} \times \mathbb{Y} \to [0, 1]\) satisfies:
\[\sum_{x \in \mathbb{X}} \sum_{y \in \mathbb{Y}} \mathbb{P}(x, y) = 1\]
For two variables, the joint probabilities can be laid out in a matrix or table. Summing across a row or column collapses that variable away, a process called marginalization:
\[\mathbb{P}(x) = \sum_{y \in \mathbb{Y}} \mathbb{P}(x, y), \qquad \mathbb{P}(y) = \sum_{x \in \mathbb{X}} \mathbb{P}(x, y)\]
The resulting distributions \(\mathbb{P}(x)\) and \(\mathbb{P}(y)\) are the marginal distributions of \(X\) and \(Y\).
Example 30 (Joint probabilities and marginalization) Consider an example adapted from Brian Hutchinson’s notes [15]. Let \(m \in \{0, 1\}\) indicate whether a meteorite hits your house on a given day (\(m = 1\)), and let \(d \in \{0, 1\}\) indicate whether you have a good day (\(d = 1\)). In this distribution, 9 out of 10 days are good days (\(\mathbb{P}(d=1) = 0.9\)), and the probability of a meteorite strike is 1 in a million (\(\mathbb{P}(m=1) = 10^{-6}\)):
Summing each row yields the marginal distribution of \(m\); summing each column yields the marginal distribution of \(d\). Conditioning inverts the perspective:
\[ \begin{aligned} \mathbb{P}(d=0 \mid m=1) &= \frac{\mathbb{P}(m=1, d=0)}{\mathbb{P}(m=1)} \\ &= \frac{0.00000098}{0.000001} \\ &= 0.98 \end{aligned} \]
Given that a meteorite struck your house, you have a 98% chance of having a bad day. In reverse:
\[ \begin{aligned} \mathbb{P}(m=1 \mid d=0) &= \frac{\mathbb{P}(m=1, d=0)}{\mathbb{P}(d=0)} \\ &= \frac{0.00000098}{0.1} \\ &= 0.0000098 \end{aligned} \]
Given that you are having a bad day, the probability of a meteorite hit is roughly 1 in 100,000. Bad days are common (\(\mathbb{P}(d=0) = 0.1\)), so having one is very weak evidence that an astronomical rarity occurred.
1.0.2 Continuous distributions and densities
When a random variable \(X\) takes values in a continuous space \(\mathcal{R}(X) \subseteq \mathbb{R}\), its behavior is described by a probability density function (PDF) \(\mathbb{p}(x) : \mathcal{R}(X) \to \mathbb{R}_+\) satisfying:
\[\int_{\mathcal{R}(X)} \mathbb{p}(x)\,\mathrm{d}x = 1\]
Probabilities are assigned to subsets \(A \subseteq \mathcal{R}(X)\) by integrating the density over that set:
\[\Pr(X \in A) = \int_{A} \mathbb{p}(x)\,\mathrm{d}x\]
Theorem 16 (Points have zero probability in continuous distributions) For any continuous random variable and any specific value \(a \in \mathbb{R}\):
\[\Pr(X = a) = 0\]
Why? As Brian Hutchinson notes [15], what is the probability that someone’s height is exactly \(6.0000000000\dots\) feet? Zero. A single real point has width zero, so the integral over a single point is zero.
Non-zero probabilities attach to intervals or regions of non-zero width:
\[\Pr(6 - \varepsilon\le X \le 6 + \varepsilon) = \int_{6-\varepsilon}^{6+\varepsilon} \mathbb{p}(x)\,\mathrm{d}x > 0 \qquad (\text{for } \varepsilon> 0)\]
Every rule developed for discrete variables carries over to continuous variables by replacing sums \(\sum_{x \in \mathcal{R}(X)}\) with integrals \(\int_{\mathcal{R}(X)} \mathrm{d}x\). For example, marginalizing out \(X\) from a joint density \(\mathbb{p}(x, y)\) to find the marginal density \(\mathbb{p}(y)\) becomes:
\[\mathbb{p}(y) = \int_{\mathcal{R}(X)} \mathbb{p}(x, y)\,\mathrm{d}x\]
Definition 24 (Normalizing constant of a density) Let \(g\) be a kernel of a density \(f\). The normalizing constant of \(g\) is the constant \(c > 0\) with
\[f(x) = c\, g(x) \quad \text{for every } x.\]
Theorem 17 (The normalizing constant makes the density integrate to 1) If \(g\) is a kernel of a density \(f\), its normalizing constant is
\[c = \frac{1}{\int_{-\infty}^{\infty} g(x)\,dx}.\]
Proof. A density integrates to 1 over the whole line, so
\[ \begin{aligned} 1 &= \int_{-\infty}^{\infty} f(x)\,dx && \text{(a density integrates to 1)} \\ &= \int_{-\infty}^{\infty} c\, g(x)\,dx && \text{(@def-normalizing-constant)} \\ &= c \int_{-\infty}^{\infty} g(x)\,dx. && \text{(constant factor out of the integral)} \end{aligned} \]
Dividing by \(\int g(x)\,dx\) gives the formula.
Definition 25 (Mode of a density or mass function) Let \(X\) be a random variable with density or mass function \(f\). The modes of \(f\) are the elements of the argmax of \(f\) over the range of \(X\):
\[{\operatorname{mode}\mathopen{}\left(f\right)\mathclose{}} = \arg \max_{x \in \mathcal{R}(X)} f(x).\]
A mode of \(f\) is an element of \({\operatorname{mode}\mathopen{}\left(f\right)\mathclose{}}\).
Remark 3 (A mode belongs to a density, not to a random variable). A mode is a property of a density or mass function, so “the mode of \(X\)” is shorthand for the mode of a chosen density of \(X\).
Two densities of the same random variable can differ on a set of probability zero, and then they can have different modes. A change of variables also moves the mode, because the density of a transformed variable includes a Jacobian factor.
The expectation of \(X\) is different: it depends only on the distribution of \(X\), so every density of \(X\) gives the same value.
Theorem 18 (A density and its kernel have the same modes) If \(g\) is a kernel of the density \(f\) of a random variable \(X\), then the modes of \(f\) are the elements of the argmax of \(g\) over the range of \(X\):
\[{\operatorname{mode}\mathopen{}\left(f\right)\mathclose{}} = \arg \max_{x \in \mathcal{R}(X)} g(x).\]
Proof. Write \(f(x) = c\, g(x)\) with \(c > 0\) (Definition 23). By the theorem that a positive multiple has the same argmax, \(\arg \max_{x \in \mathcal{R}(X)} f(x) = \arg \max_{x \in \mathcal{R}(X)} g(x)\). The left side is \({\operatorname{mode}\mathopen{}\left(f\right)\mathclose{}}\) by Definition 25.
Example 31 (The kernel, normalizing constant, and mode of an exponential density) The exponential density with rate \({\lambda}\) is \(f(t) = {\lambda}\text{e}^{-{\lambda}t}\) for \(t \ge 0\), and \(f(t) = 0\) for \(t < 0\). Its kernel is \(g(t) = \text{e}^{-{\lambda}t}\) for \(t \ge 0\), and \(g(t) = 0\) for \(t < 0\), and the normalizing constant is
\[ \begin{aligned} c &= \frac{1}{\int_0^{\infty} \text{e}^{-{\lambda}t}\,dt} && \text{(@thm-normalizing-constant-formula)} \\ &= \frac{1}{\mathopen{}\left[-\text{e}^{-{\lambda}t}/{\lambda}\right]\mathclose{}_0^{\infty}} && \text{(antiderivative of } \text{e}^{-{\lambda}t} \text{)} \\ &= \frac{1}{1/{\lambda}} && \text{(evaluate at } \infty \text{ and } 0 \text{)} \\ &= {\lambda}. && \text{(simplify)} \end{aligned} \]
The kernel is largest at \(t = 0\), so by Theorem 18 the mode of \(f\) is \(0\).
Hutchinson’s Probability Refresher (27 min) covers probability mass functions and probability density functions [16]. The login for the video site is posted on Canvas.
5MinuteAI explains discrete random variables and continuous random variables in five minutes each, following the University of Washington’s CSE 312 [17].
2 Survival, hazard, and cumulative hazard functions
Definition 26 (Survival function (survivor function)) The survival function (or survivor function) of a random variable \(T\), denoted \(\mathbb{S}(t)\), is the probability that \(T\) exceeds \(t\) [18]:
\[\mathbb{S}(t) \stackrel{\text{def}}{=}\Pr(T > t)\]
Remark. The name comes from time-to-event analysis: if \(T\) is the time at which a participant dies, then \(\mathbb{S}(t)\) is the probability that the participant is still alive at time \(t\). The definition itself applies to any random variable.
Proof. The event \(\{T > t\}\) is the complement of the event \(\{T \le t\}\), so the complement rule gives:
\[ \begin{aligned} \mathbb{S}(t) &\stackrel{\text{def}}{=}\Pr(T > t) && \text{(definition of the survival function)} \\ &= 1 - \Pr(T \le t) && \text{(complement rule)} \\ &= 1 - F(t) && \text{(definition of the CDF)} \end{aligned} \]
For continuous \(T\), the density integrates to 1 over the real line (Theorem 11), and \(F\) is the integral of the density up to \(t\) (Theorem 7), so:
\[ \begin{aligned} 1 - F(t) &= \int_{u=-\infty}^{\infty} f(u)\,du - \int_{u=-\infty}^{t} f(u)\,du && \text{(total integral is 1; } F \text{ is the integral of } f \text{)} \\ &= \int_{u=t}^{\infty} f(u)\,du && \text{(split the first integral at } t \text{ and cancel)} \end{aligned} \]
Example 32 (Survival function of an exponential distribution) Let \(T\) be exponential with rate \({\lambda}> 0\) (Definition 14), so \(T\) has density \(f(t) = {\lambda}\text{e}^{-{\lambda}t}\) for \(t \ge 0\). For \(t \ge 0\), by Theorem 19:
\[ \begin{aligned} \mathbb{S}(t) &= \int_{u=t}^{\infty} {\lambda}\text{e}^{-{\lambda}u}\,du && \text{(integral form of the survival function)} \\ &= \mathopen{}\left[-\text{e}^{-{\lambda}u}\right]\mathclose{}_{u=t}^{\infty} && \text{(antiderivative of } {\lambda}\text{e}^{-{\lambda}u} \text{)} \\ &= 0 - \mathopen{}\left(-\text{e}^{-{\lambda}t}\right)\mathclose{} && \text{(evaluate at the bounds; } \text{e}^{-{\lambda}u} \to 0 \text{ as } u \to \infty \text{)} \\ &= \text{e}^{-{\lambda}t} && \text{(simplify)} \end{aligned} \]
For \(t < 0\),
\[ \begin{aligned} \mathbb{S}(t) &= \Pr(T > t) \\ &= 1, \end{aligned} \]
since \(T \ge 0\). With \({\lambda}= 0.5\), for example, \(\mathbb{S}(2) = \text{e}^{-1} \approx 0.368\).
Definition 27 (Hazard function (hazard rate, hazard rate function)) The hazard function (also called the hazard rate or hazard rate function) of a continuous random variable \(T\) at a value \(t\) with \(\Pr(T \ge t) > 0\), typically denoted \({\lambda}(t)\) or \(\mathbb{h}(t)\), is the limit of the conditional probability that \(T\) falls in an interval starting at \(t\), given \(T \ge t\), divided by the width of that interval, as that width shrinks to 0 [19]:
\[{\lambda}(t) \stackrel{\text{def}}{=}\lim_{\Delta\downarrow 0} \frac{\Pr(t \le T < t + \Delta\mid T \ge t)}{\Delta}\]
Remark. Sources differ on the symbol: \(\mathbb{h}(t)\) appears in Dobson and Barnett [20], Vittinghoff et al. [21], Klein and Moeschberger [22], and Kleinbaum and Klein [23], while \(\lambda(t)\) appears in Rothman et al. [12] and Kalbfleisch and Prentice [24].
If \(T\) is the time at which an event occurs, then \({\lambda}(t)\) is a rate, not a probability: for a small interval width \(\Delta> 0\), \({\lambda}(t) \cdot\Delta\) is approximately the probability that the event occurs in \([t, t + \Delta)\), given that it has not occurred before \(t\). Many sources write the hazard as \({\lambda}(t) = \mathbb{p}(T = t \mid T \ge t)\), reading it as the density of \(T\) at \(t\), conditional on the event \(T \ge t\); that notation abbreviates the limit in Definition 27. For a discrete \(T\), the analogous quantity is the discrete-time hazard.
The name “hazard” carries a connotation that the event is undesirable — death, relapse, equipment failure, and so on. When the event in question is neutral or desirable (recovery, conception, graduation, response to treatment), the same quantity \({\lambda}(t)\) is often called the event incidence rate instead. This terminology parallels the convention that conditional probabilities of undesirable events are called risks, while the same conditional probabilities for neutral or desirable events are simply called probabilities. The math is identical; only the name changes with the valence of the event.
Example 33 (A hazard greater than 1) A hazard can exceed 1, just as a density can (Example 12). Let \(T\) be exponential with rate \({\lambda}= 2\), so \(\mathbb{S}(t) = \text{e}^{-2t}\) for \(t \ge 0\) (Example 32). Since \(T\) is continuous, \(\Pr(T = s) = 0\) for every \(s\), so
\[ \begin{aligned} \Pr(T \ge s) &= \Pr(T > s) \\ &= \mathbb{S}(s). \end{aligned} \]
For \(t \ge 0\) and \(\Delta> 0\), \(\{T \ge t\}\) is the disjoint union of \(\{t \le T < t + \Delta\}\) and \(\{T \ge t + \Delta\}\), so:
\[ \begin{aligned} \Pr(t \le T < t + \Delta\mid T \ge t) &= \frac{\Pr(\{t \le T < t + \Delta\} \cap \{T \ge t\})}{\Pr(T \ge t)} && \text{(definition of conditional probability)} \\ &= \frac{\Pr(t \le T < t + \Delta)}{\Pr(T \ge t)} && \text{(subset property)} \\ &= \frac{\Pr(T \ge t) - \Pr(T \ge t + \Delta)}{\Pr(T \ge t)} && \text{(additivity)} \\ &= \frac{\mathbb{S}(t) - \mathbb{S}(t + \Delta)}{\mathbb{S}(t)} && \text{(} \Pr(T \ge s) = \mathbb{S}(s) \text{)} \\ &= \frac{\text{e}^{-2t} - \text{e}^{-2(t + \Delta)}}{\text{e}^{-2t}} && \text{(exponential survival function)} \\ &= 1 - \text{e}^{-2\Delta} && \text{(divide by } \text{e}^{-2t} \text{)} \end{aligned} \]
The subset property applies because \(\{t \le T < t + \Delta\} \subseteq \{T \ge t\}\). Then:
\[ \begin{aligned} {\lambda}(t) &= \lim_{\Delta\downarrow 0} \frac{1 - \text{e}^{-2\Delta}}{\Delta} && \text{(definition of the hazard function)} \\ &= \left.{\frac{\partial}{\partial \Delta} \mathopen{}\left(1 - \text{e}^{-2\Delta}\right)\mathclose{}}\right|_{\Delta= 0} && \text{(definition of the derivative; } 1 - \text{e}^{0} = 0 \text{)} \\ &= 2\text{e}^{0} && \text{(chain rule)} \\ &= 2 && \text{(} \text{e}^{0} = 1 \text{)} \end{aligned} \]
So \({\lambda}(t) = 2 > 1\) at every \(t \ge 0\): a hazard is a rate, not a probability.
Definition 28 (Discrete-time hazard) The discrete-time hazard of a discrete random variable \(T\) at a value \(t\) with \(\Pr(T \ge t) > 0\) is the conditional probability that \(T\) equals \(t\), given \(T \ge t\):
\[\Pr(T = t \mid T \ge t)\]
Example 34 (Rolling until the first six) Roll a fair die repeatedly, and let \(T\) be the number of the roll that first shows a six. Then \(T \ge t\) exactly when the first \(t - 1\) rolls are not sixes, which has probability \((5/6)^{t-1}\), and \(T = t\) when, in addition, roll \(t\) is a six. So, for each \(t = 1, 2, \ldots\):
\[ \begin{aligned} \Pr(T = t \mid T \ge t) &= \frac{\Pr(T = t,\, T \ge t)}{\Pr(T \ge t)} && \text{(definition of conditional probability)} \\ &= \frac{\Pr(T = t)}{\Pr(T \ge t)} && \text{(} T = t \text{ implies } T \ge t \text{)} \\ &= \frac{(5/6)^{t-1} \cdot(1/6)}{(5/6)^{t-1}} && \text{(rolls are independent)} \\ &= \frac{1}{6} && \text{(cancel)} \end{aligned} \]
The discrete-time hazard is the same at every roll: having gone without a six so far does not change the chance of a six on the next roll.
Corollary 3 (A discrete-time hazard lies in \({[0, 1]}\)) The discrete-time hazard of a discrete random variable \(T\) at a value \(t\) with \(\Pr(T \ge t) > 0\) satisfies:
\[0 \le \Pr(T = t \mid T \ge t) \le 1\]
Proof. \(\{T \ge t\}\) is the disjoint union of \(\{T = t\}\) and \(\{T > t\}\), so, by additivity, \(\Pr(T \ge t) = \Pr(T = t) + \Pr(T > t) \ge \Pr(T = t)\). Since \(\{T = t\} \subseteq \{T \ge t\}\):
\[ \begin{aligned} \Pr(T = t \mid T \ge t) &= \frac{\Pr(\{T = t\} \cap \{T \ge t\})}{\Pr(T \ge t)} && \text{(definition of conditional probability)} \\ &= \frac{\Pr(T = t)}{\Pr(T \ge t)} && \text{(subset property)} \end{aligned} \]
The numerator is non-negative and at most the positive denominator, so the ratio lies in \([0, 1]\).
Remark. Unlike the continuous-time hazard function, which is a rate and can exceed 1 (Example 33), the discrete-time hazard is a probability.
Theorem 20 (A hazard is non-negative) If \({\lambda}(t)\) is the hazard function of a continuous random variable \(T\) at a value \(t\) with \(\Pr(T \ge t) > 0\), then:
\[{\lambda}(t) \ge 0\]
Proof. For each \(\Delta> 0\), the quantity that defines the hazard is non-negative:
\[ \begin{aligned} \frac{\Pr(t \le T < t + \Delta\mid T \ge t)}{\Delta} &\ge 0 && \text{(a conditional probability is non-negative, and } \Delta> 0 \text{)} \end{aligned} \]
A limit of non-negative values is non-negative, so \({\lambda}(t) \ge 0\).
Remark 4 (The range of a hazard). A hazard can exceed \(1\): Example 33 has \({\lambda}(t) = 2\). Together with Theorem 20, the hazards of a continuous random variable take values in \([0, \infty)\). The discrete-time hazard differs: it lies in \([0, 1]\) by Corollary 3.
Theorem 21 (Hazard equals density over survival) If \(T\) is a continuous random variable with density \(f(t)\) and survival function \(\mathbb{S}(t)\), then for every \(t\) with \(\mathbb{S}(t) > 0\) at which \(f\) is continuous:
\[{\lambda}(t) = \frac{f(t)}{\mathbb{S}(t)}\]
Proof. The proof uses three facts. First, for \(\Delta> 0\) the event \(\{t \le T < t + \Delta\}\) is a subset of the event \(\{T \ge t\}\), so intersecting them leaves \(\{t \le T < t + \Delta\}\) (the subset property). Second, \(\Pr(T = t) = 0\) for a continuous \(T\), so
\[ \begin{aligned} \Pr(T \ge t) &= \Pr(T > t) \\ &= \mathbb{S}(t), \end{aligned} \]
which is positive. Third, because \(f\) is continuous at \(t\), Theorem 10 gives \(f(t)\) as the limit of \(\Pr(t \le T < t + \Delta) / \Delta\).
\[ \begin{aligned} {\lambda}(t) &\stackrel{\text{def}}{=}\lim_{\Delta\downarrow 0} \frac{\Pr(t \le T < t + \Delta\mid T \ge t)}{\Delta} && \text{(definition of the hazard function)} \\ &= \lim_{\Delta\downarrow 0} \frac{1}{\Delta} \cdot\frac{\Pr(\{t \le T < t + \Delta\} \cap \{T \ge t\})}{\Pr(T \ge t)} && \text{(definition of conditional probability)} \\ &= \lim_{\Delta\downarrow 0} \frac{1}{\Delta} \cdot\frac{\Pr(t \le T < t + \Delta)}{\Pr(T \ge t)} && \text{(subset property)} \\ &= \frac{1}{\Pr(T \ge t)} \cdot\lim_{\Delta\downarrow 0} \frac{\Pr(t \le T < t + \Delta)}{\Delta} && \text{(} \Pr(T \ge t) \text{ does not depend on } \Delta\text{)} \\ &= \frac{f(t)}{\Pr(T \ge t)} && \text{(density as a limit; } f \text{ is continuous at } t \text{)} \\ &= \frac{f(t)}{\mathbb{S}(t)} && \text{(} \Pr(T = t) = 0 \text{ for continuous } T \text{)} \end{aligned} \]
Example 35 (Hazard function of an exponential distribution) Continuing Example 32, for \(t > 0\), where \(f\) is continuous:
\[ \begin{aligned} {\lambda}(t) &= \frac{f(t)}{\mathbb{S}(t)} && \text{(hazard equals density over survival)} \\ &= \frac{{\lambda}\text{e}^{-{\lambda}t}}{\text{e}^{-{\lambda}t}} && \text{(substitute the exponential density and survival function)} \\ &= {\lambda} && \text{(cancel } \text{e}^{-{\lambda}t} \text{)} \end{aligned} \]
At \(t = 0\), where \(f\) jumps from \(0\) to \({\lambda}\), Definition 27 gives the same value directly: \(\Pr(0 \le T < \Delta\mid T \ge 0) / \Delta= \mathopen{}\left(1 - \text{e}^{-{\lambda}\Delta}\right)\mathclose{} / \Delta\to {\lambda}\) as \(\Delta\downarrow 0\). So the exponential distribution has a constant hazard for \(t \ge 0\), equal to its rate parameter; for \(t < 0\), \(f(t) = 0\), so \({\lambda}(t) = 0\).
Definition 29 (Cumulative hazard function) The cumulative hazard function of a continuous random variable \(T\), often denoted \({\Lambda}(t)\) or \(\mathbb{H}(t)\), is the integral of its hazard function up to \(t\):
\[{\Lambda}(t) \stackrel{\text{def}}{=}\int_{u=-\infty}^{t} {\lambda}(u)\,du\]
Example 36 (Cumulative hazard function of an exponential distribution) Continuing Example 35, the hazard is \({\lambda}(u) = {\lambda}\) for \(u \ge 0\) and \(0\) for \(u < 0\), so for \(t \ge 0\):
\[ \begin{aligned} {\Lambda}(t) &= \int_{u=-\infty}^{0} 0\,du + \int_{u=0}^{t} {\lambda}\,du && \text{(split the integral at } 0 \text{)} \\ &= 0 + {\lambda}t && \text{(integrate each piece)} \\ &= {\lambda}t && \text{(simplify)} \end{aligned} \]
and \({\Lambda}(t) = 0\) for \(t < 0\).
Corollary 4 (Cumulative hazard of a non-negative random variable) If \(T\) is a continuous random variable with \(T \ge 0\), such as a time to event, then \({\lambda}(u) = 0\) for every \(u < 0\), and for every \(t \ge 0\):
\[{\Lambda}(t) = \int_{u=0}^{t} {\lambda}(u)\,du\]
Proof. Let \(u < 0\). Every outcome has \(T(\omega) \ge 0 > u\), so \(\{T \ge u\} = \Omega\), which has probability \(1 > 0\), and \({\lambda}(u)\) is defined. For \(0 < \Delta\le -u\), the event \(\{u \le T < u + \Delta\}\) is empty, because \(u + \Delta\le 0\), so:
\[ \begin{aligned} {\lambda}(u) &= \lim_{\Delta\downarrow 0} \frac{\Pr(u \le T < u + \Delta\mid T \ge u)}{\Delta} && \text{(definition of the hazard function)} \\ &= \lim_{\Delta\downarrow 0} \frac{\Pr(\{u \le T < u + \Delta\} \cap \{T \ge u\})}{\Delta\cdot\Pr(T \ge u)} && \text{(definition of conditional probability)} \\ &= \lim_{\Delta\downarrow 0} \frac{\Pr(\emptyset \cap \Omega)}{\Delta\cdot\Pr(\Omega)} && \text{(the two events found above)} \\ &= \lim_{\Delta\downarrow 0} \frac{0}{\Delta\cdot 1} && \text{(} \emptyset \cap \Omega= \emptyset \text{; } \Pr(\emptyset) = 0 \text{; } \Pr(\Omega) = 1 \text{)} \\ &= 0 && \text{(simplify)} \end{aligned} \]
Then, for \(t \ge 0\):
\[ \begin{aligned} {\Lambda}(t) &= \int_{u=-\infty}^{t} {\lambda}(u)\,du && \text{(definition of the cumulative hazard)} \\ &= \int_{u=-\infty}^{0} {\lambda}(u)\,du + \int_{u=0}^{t} {\lambda}(u)\,du && \text{(split the integral at } 0 \text{)} \\ &= \int_{u=-\infty}^{0} 0\,du + \int_{u=0}^{t} {\lambda}(u)\,du && \text{(} {\lambda}(u) = 0 \text{ for } u < 0 \text{)} \\ &= \int_{u=0}^{t} {\lambda}(u)\,du && \text{(integrate)} \end{aligned} \]
Remark. The form in Corollary 4, with lower limit \(0\), is the form most survival-analysis texts use.
Corollary 5 (Survival function from the cumulative hazard) If \(T\) is a continuous random variable whose density \(f\) is continuous at all but finitely many points, then for every \(t\) with \(\mathbb{S}(t) > 0\):
\[{\Lambda}(t) = -{\operatorname{log}\mathopen{}\left\{\mathbb{S}(t)\right\}\mathclose{}}\]
and therefore:
\[\mathbb{S}(t) = {\operatorname{exp}\mathopen{}\left\{-{\Lambda}(t)\right\}\mathclose{}} \tag{1}\]
Proof. Since \(\mathbb{S}(t) = 1 - F(t)\) (Theorem 19) and \(F\) has derivative \(f\) wherever \(f\) is continuous (Theorem 8), \(\mathbb{S}\) has derivative \(-f\) at those points. At each such \(u\) with \(\mathbb{S}(u) > 0\):
\[ \begin{aligned} \frac{d}{du}\mathopen{}\left(-{\operatorname{log}\mathopen{}\left\{\mathbb{S}(u)\right\}\mathclose{}}\right)\mathclose{} &= -\frac{1}{\mathbb{S}(u)} \cdot\frac{d}{du}\mathbb{S}(u) && \text{(chain rule)} \\ &= -\frac{1}{\mathbb{S}(u)} \cdot\mathopen{}\left(-f(u)\right)\mathclose{} && \text{(} \mathbb{S}' = -f \text{)} \\ &= \frac{f(u)}{\mathbb{S}(u)} && \text{(simplify)} \\ &= {\lambda}(u) && \text{(hazard equals density over survival)} \end{aligned} \]
The function \(-{\operatorname{log}\mathopen{}\left\{\mathbb{S}(u)\right\}\mathclose{}}\) is continuous, because \(\mathbb{S}\) is continuous for a continuous \(T\), and it has derivative \({\lambda}(u)\) at all but finitely many points, so it is an antiderivative of \({\lambda}\) for the fundamental theorem of calculus. As \(u \to -\infty\), \(\mathbb{S}(u) \to 1\), so \(-{\operatorname{log}\mathopen{}\left\{\mathbb{S}(u)\right\}\mathclose{}} \to 0\). Therefore:
\[ \begin{aligned} {\Lambda}(t) &\stackrel{\text{def}}{=}\int_{u=-\infty}^{t} {\lambda}(u)\,du && \text{(definition of the cumulative hazard)} \\ &= \mathopen{}\left[-{\operatorname{log}\mathopen{}\left\{\mathbb{S}(u)\right\}\mathclose{}}\right]\mathclose{}_{u=-\infty}^{t} && \text{(} -\log \mathbb{S}\text{ is an antiderivative of } {\lambda}\text{)} \\ &= -{\operatorname{log}\mathopen{}\left\{\mathbb{S}(t)\right\}\mathclose{}} - 0 && \text{(evaluate at the bounds)} \\ &= -{\operatorname{log}\mathopen{}\left\{\mathbb{S}(t)\right\}\mathclose{}} && \text{(simplify)} \end{aligned} \]
Exponentiating both sides of \(-{\Lambda}(t) = {\operatorname{log}\mathopen{}\left\{\mathbb{S}(t)\right\}\mathclose{}}\) gives Equation 1.
Example 37 (Recovering the exponential survival function from its cumulative hazard) Continuing Example 36, for \(t \ge 0\):
\[ \begin{aligned} \mathbb{S}(t) &= {\operatorname{exp}\mathopen{}\left\{-{\Lambda}(t)\right\}\mathclose{}} && \text{(survival function from the cumulative hazard)} \\ &= {\operatorname{exp}\mathopen{}\left\{-{\lambda}t\right\}\mathclose{}} && \text{(substitute } {\Lambda}(t) = {\lambda}t \text{)} \end{aligned} \]
which matches the survival function computed directly in Example 32.
Remark. For a continuous random variable \(T \ge 0\) whose density is continuous at all but finitely many points, the following results connect the functions in Table 2:
Each arrow in the following diagram converts one function into the next:
\[ f(t) \xleftarrow[\mathbb{S}(t){\lambda}(t)]{-\mathbb{S}'(t)} \mathbb{S}(t) \xleftarrow[]{{\operatorname{exp}\mathopen{}\left\{-{\Lambda}(t)\right\}\mathclose{}}} {\Lambda}(t) \xleftarrow[]{\int_{u=0}^t {\lambda}(u)\,du} {\lambda}(t) \xleftarrow[]{{\operatorname{exp}\mathopen{}\left\{\eta(t)\right\}\mathclose{}}} \eta(t) \]
\[ f(t) \xrightarrow[\int_{u=t}^\infty f(u)\,du]{f(t)/{\lambda}(t)} \mathbb{S}(t) \xrightarrow[-{\operatorname{log}\mathopen{}\left\{\mathbb{S}(t)\right\}\mathclose{}}]{} {\Lambda}(t) \xrightarrow[{\Lambda}'(t)]{} {\lambda}(t) \xrightarrow[{\operatorname{log}\mathopen{}\left\{{\lambda}(t)\right\}\mathclose{}}]{} \eta(t) \]
3 Further reading
Evans and Rosenthal’s free textbook [25] covers random variables in Chapter 2.
Definitions on this page
- Definition 1 (Random variable)
- Definition 2 (Range of a random variable)
- Definition 3 (Discrete random variable)
- Definition 4 (Continuous random variable)
- Definition 5 (Realization (observed value))
- Definition 6 (Probability mass function (PMF))
- Definition 7 (Bernoulli distribution)
- Definition 8 (Probability density function (PDF))
- Definition 9 (Uniform distribution on a set)
- Definition 10 (Uniform distribution on an interval)
- Definition 11 (Density with respect to a measure, Radon–Nikodym derivative)
- Definition 12 (Normal distribution (Gaussian distribution))
- Definition 13 (Standard normal distribution)
- Definition 14 (Exponential distribution)
- Definition 15 (Cumulative distribution function (CDF))
- Definition 16 (Quantile function (population inverse CDF))
- Definition 17 (Jointly distributed random variables)
- Definition 18 (Joint distribution)
- Definition 19 (Joint probability mass function (joint PMF))
- Definition 20 (Marginal distribution)
- Definition 21 (Joint probability density function (joint density))
- Definition 22 (Joint density-mass function)
- Definition 23 (Kernel of a density)
- Definition 24 (Normalizing constant of a density)
- Definition 25 (Mode of a density or mass function)
- Definition 26 (Survival function (survivor function))
- Definition 27 (Hazard function (hazard rate, hazard rate function))
- Definition 28 (Discrete-time hazard)
- Definition 29 (Cumulative hazard function)