Random variables
1 Random variables
2 Characteristics of probability distributions
Proof. For \(a \le b\), the difference \(g - f\) is \(0\) at all but finitely many points, so:
\[ \begin{aligned} \int_a^b g(x)\,dx &= \int_a^b f(x)\,dx + \int_a^b \mathopen{}\left(g(x) - f(x)\right)\mathclose{}\,dx && \text{(linearity of the integral)} \\ &= \int_a^b f(x)\,dx + 0 && \text{(a function that is 0 at all but finitely many points integrates to 0)} \\ &= \Pr(a \le X \le b) && \text{(} f \text{ is a density of } X \text{)} \end{aligned} \]
Together with \(g \ge 0\), this is Definition 8 for \(g\).
Proof. Non-decreasing. For \(s < t\), \(\{X \le t\}\) is the disjoint union of \(\{X \le s\}\) and \(\{s < X \le t\}\), so finite additivity gives:
\[ \begin{aligned} F(t) &= \Pr(X \le s) + \Pr(s < X \le t) && \text{(additivity, and the definition of the CDF)} \\ &\ge \Pr(X \le s) && \text{(probabilities are non-negative)} \\ &= F(s) && \text{(definition of the CDF)} \end{aligned} \]
Limit at \(-\infty\). The events \(A_n = \{X \le -n\}\), \(n = 1, 2, \ldots\), are decreasing, and their intersection is empty, because every outcome \(\omega\) has \(X(\omega) > -n\) once \(n > -X(\omega)\). By continuity of probability, \(F(-n) = \Pr(A_n) \to 0\). For \(t \le -n\), \(0 \le F(t) \le F(-n)\), because \(F\) is non-decreasing, so \(F(t) \to 0\) as \(t \to -\infty\).
Limit at \(\infty\). The events \(B_n = \{X > n\}\) are decreasing, and their intersection is empty, because every outcome \(\omega\) has \(X(\omega) \le n\) once \(n \ge X(\omega)\). By continuity of probability, \(\Pr(B_n) \to 0\), so, by the complement rule:
\[ \begin{aligned} F(n) &= 1 - \Pr(X > n) && \text{(complement rule)} \\ &\to 1 - 0 && \text{(} \Pr(B_n) \to 0 \text{)} \end{aligned} \]
For \(t \ge n\), \(F(n) \le F(t) \le 1\), because \(F\) is non-decreasing, so \(F(t) \to 1\) as \(t \to \infty\).
Proof. For any \(a < t\), \(\{X \le t\}\) is the disjoint union of \(\{X \le a\}\) and \(\{a < X \le t\}\), and \(\Pr(X = a) = 0\) because \(X\) is continuous. So:
\[ \begin{aligned} F(t) &= \Pr(X \le a) + \Pr(a < X \le t) && \text{(additivity, and the definition of the CDF)} \\ &= F(a) + \Pr(a \le X \le t) && \text{(definition of the CDF; } \Pr(X = a) = 0 \text{)} \\ &= F(a) + \int_a^t f(x)\,dx && \text{(definition of the density)} \end{aligned} \]
Letting \(a \to -\infty\), \(F(a) \to 0\) (Theorem 4), which gives \(F(t) = \int_{-\infty}^{t} f(x)\,dx\). The fundamental theorem of calculus then gives \(\frac{\partial}{\partial t} \int_{-\infty}^{t} f(x)\,dx = f(t)\) at every \(t\) where \(f\) is continuous.
Proof. At every point \(t\) where \(f\) is continuous, Theorem 6 gives \(F'(t) = f(t)\), and by assumption those are all but finitely many points.
Proof. For events \(A \subseteq B\), \(B\) is the disjoint union of \(A\) and \(B \setminus A\), so additivity gives \(\Pr(A) \le \Pr(B)\). For any \(x\) and \(\Delta > 0\), \(\{X = x\} \subseteq \{x - \Delta < X \le x\}\), so:
\[ \begin{aligned} 0 &\le \Pr(X = x) && \text{(probabilities are non-negative)} \\ &\le \Pr(x - \Delta < X \le x) && \text{(} \{X = x\} \subseteq \{x - \Delta < X \le x\} \text{)} \\ &= F(x) - F(x - \Delta) && \text{(additivity, and the definition of the CDF)} \end{aligned} \]
and \(F(x) - F(x - \Delta) \to 0\) as \(\Delta \downarrow 0\), because \(F\) is continuous. So \(\Pr(X = x) = 0\) for every \(x\), and \(X\) is continuous. Then, for \(a \le b\):
\[ \begin{aligned} \Pr(a \le X \le b) &= \Pr(a < X \le b) && \text{(} \Pr(X = a) = 0 \text{)} \\ &= F(b) - F(a) && \text{(additivity, and the definition of the CDF)} \\ &= \int_a^b F'(x)\,dx && \text{(fundamental theorem of calculus)} \end{aligned} \]
The last step applies the fundamental theorem of calculus on each piece of \([a, b]\) between exceptional points, where \(F'\) is continuous, and joins the pieces using the continuity of \(F\) at the exceptional points. Finally, \(F' \ge 0\) wherever it exists, because \(F\) is non-decreasing (Theorem 4).
Proof. For \(\Delta > 0\):
\[ \begin{aligned} \frac{\Pr(x \le X < x + \Delta)}{\Delta} &= \frac{\Pr(x < X \le x + \Delta)}{\Delta} && \text{(} \Pr(X = x) = \Pr(X = x + \Delta) = 0 \text{)} \\ &= \frac{F(x + \Delta) - F(x)}{\Delta} && \text{(additivity, and the definition of the CDF)} \end{aligned} \]
By Theorem 6, \(F\) is differentiable at \(x\) with \(F'(x) = f(x)\), so this difference quotient converges to \(f(x)\) as \(\Delta \downarrow 0\).
Proof. The first two lines of the proof of Theorem 8 use only that \(X\) is continuous, so for every \(\Delta > 0\) the two quotients are equal. Equal functions of \(\Delta\) have the same limit as \(\Delta \downarrow 0\), or both have none.
Proof. \[ \begin{aligned} \int_{-\infty}^{\infty} f(x)\, dx &= \lim_{b \to \infty} \int_{-\infty}^{b} f(x)\, dx && \text{(definition of an improper integral)} \\ &= \lim_{b \to \infty} F(b) && \text{(the CDF is the integral of the density)} \\ &= 1 && \text{(limit of a CDF)} \end{aligned} \]
The last step is Theorem 4.
Proof. The event \(\{X = x\}\) is the disjoint union of the events \(\{X = x\} \cap \{Y = y\}\) over the countably many values \(y \in \mathcal{R}(Y)\), so:
\[ \begin{aligned} \operatorname{P}(X = x) &= \Pr\mathopen{}\left(\bigcup_{y \in \mathcal{R}(Y)} \mathopen{}\left(\{X = x\} \cap \{Y = y\}\right)\mathclose{}\right)\mathclose{} && \text{(the events partition } \{X = x\} \text{)} \\ &= \sum_{y \in \mathcal{R}(Y)} \Pr(\{X = x\} \cap \{Y = y\}) && \text{(countable additivity)} \\ &= \sum_{y \in \mathcal{R}(Y)} \operatorname{P}(X = x,\, Y = y) && \text{(definition of the joint PMF)} \end{aligned} \]
Proof. Suppose \(f\) were a joint density of \((X, X)\), and let \(L = \mathopen{}\left\{(x, y) : y = x\right\}\mathclose{}\) be the diagonal line. Every outcome \(\omega\) has \(X(\omega) = X(\omega)\), so \(\mathopen{}\left\{(X, X) \in L\right\}\mathclose{} = \Omega\). For each \(x\), the function \(y \mapsto \text{1}_{y = x} f(x, y)\) is \(0\) except at the single point \(y = x\), so its integral over \(y\) is \(0\). Then:
\[ \begin{aligned} 1 &= \Pr((X, X) \in L) && \text{(} \mathopen{}\left\{(X, X) \in L\right\}\mathclose{} = \Omega \text{, and } \Pr(\Omega) = 1 \text{)} \\ &= \iint_L f(x, y)\,dx\,dy && \text{(definition of a joint density)} \\ &= \int_{-\infty}^{\infty} \mathopen{}\left(\int_{-\infty}^{\infty} \text{1}_{y = x} f(x, y)\,dy\right)\mathclose{}\,dx && \text{(iterate the integral; Tonelli's theorem)} \\ &= \int_{-\infty}^{\infty} 0\,dx && \text{(the inner integrand is 0 except at } y = x \text{)} \\ &= 0 && \text{(integrate)} \end{aligned} \]
which is a contradiction. Tonelli’s theorem allows the iterated integral because \(f \ge 0\) (Fubini–Tonelli theorem; Billingsley (1995), Theorem 18.3).
Proof. For \(a \le b\), the event \(\{a \le X \le b\}\) is the event that \((X, Y)\) falls in the strip \([a, b] \times \mathbb{R}\), so:
\[ \begin{aligned} \Pr(a \le X \le b) &= \Pr((X, Y) \in [a, b] \times \mathbb{R}) && \text{(same event)} \\ &= \iint_{[a, b] \times \mathbb{R}} f_{X,Y}(x, y)\,dx\,dy && \text{(definition of a joint density)} \\ &= \int_a^b \mathopen{}\left(\int_{-\infty}^{\infty} f_{X,Y}(x, y)\,dy\right)\mathclose{}\,dx && \text{(iterate the integral; Tonelli's theorem)} \\ &= \int_a^b f_X(x)\,dx && \text{(definition of } f_X \text{)} \end{aligned} \]
Tonelli’s theorem allows the iterated integral because \(f_{X,Y} \ge 0\) (Fubini–Tonelli theorem; Billingsley (1995), Theorem 18.3). So \(f_X\) satisfies Definition 8.
Proof. Every outcome has \(Y(\omega) \in \mathbb{R}\), so \(\mathopen{}\left\{Y \in \mathbb{R}\right\}\mathclose{} = \Omega\), and:
\[ \begin{aligned} \operatorname{P}(X = x) &= \Pr(\mathopen{}\left\{X = x\right\}\mathclose{} \cap \Omega) && \text{(} \mathopen{}\left\{X = x\right\}\mathclose{} \subseteq \Omega \text{)} \\ &= \Pr(X = x,\, Y \in \mathbb{R}) && \text{(} \mathopen{}\left\{Y \in \mathbb{R}\right\}\mathclose{} = \Omega \text{)} \\ &= \int_{-\infty}^{\infty} \operatorname{p}(X = x,\, Y = y)\,dy && \text{(definition of a joint density-mass function, with } B = \mathbb{R} \text{)} \end{aligned} \]
Hutchinson’s Probability Refresher (27 min) covers probability mass functions and probability density functions (Hutchinson, n.d.). The login for the video site is posted on Canvas.
3 Survival, hazard, and cumulative hazard functions
Proof. The event \(\{T > t\}\) is the complement of the event \(\{T \le t\}\), so the complement rule gives:
\[ \begin{aligned} \operatorname{S}(t) &\stackrel{\text{def}}{=}\Pr(T > t) && \text{(definition of the survival function)} \\ &= 1 - \Pr(T \le t) && \text{(complement rule)} \\ &= 1 - F(t) && \text{(definition of the CDF)} \end{aligned} \]
For continuous \(T\), the density integrates to 1 over the real line (Theorem 9), and \(F\) is the integral of the density up to \(t\) (Theorem 6), so:
\[ \begin{aligned} 1 - F(t) &= \int_{u=-\infty}^{\infty} f(u)\,du - \int_{u=-\infty}^{t} f(u)\,du && \text{(total integral is 1; } F \text{ is the integral of } f \text{)} \\ &= \int_{u=t}^{\infty} f(u)\,du && \text{(split the first integral at } t \text{ and cancel)} \end{aligned} \]
Proof. \(\{T \ge t\}\) is the disjoint union of \(\{T = t\}\) and \(\{T > t\}\), so, by additivity, \(\Pr(T \ge t) = \Pr(T = t) + \Pr(T > t) \ge \Pr(T = t)\). Since \(\{T = t\} \subseteq \{T \ge t\}\):
\[ \begin{aligned} \Pr(T = t \mid T \ge t) &= \frac{\Pr(\{T = t\} \cap \{T \ge t\})}{\Pr(T \ge t)} && \text{(definition of conditional probability)} \\ &= \frac{\Pr(T = t)}{\Pr(T \ge t)} && \text{(subset property)} \end{aligned} \]
The numerator is non-negative and at most the positive denominator, so the ratio lies in \([0, 1]\).
Proof. The proof uses three facts. First, for \(\Delta > 0\) the event \(\{t \le T < t + \Delta\}\) is a subset of the event \(\{T \ge t\}\), so intersecting them leaves \(\{t \le T < t + \Delta\}\) (the subset property). Second, \(\Pr(T = t) = 0\) for a continuous \(T\), so \(\Pr(T \ge t) = \Pr(T > t) = \operatorname{S}(t)\), which is positive. Third, because \(f\) is continuous at \(t\), Theorem 8 gives \(f(t)\) as the limit of \(\Pr(t \le T < t + \Delta) / \Delta\).
\[ \begin{aligned} {\lambda}(t) &\stackrel{\text{def}}{=}\lim_{\Delta \downarrow 0} \frac{\Pr(t \le T < t + \Delta \mid T \ge t)}{\Delta} && \text{(definition of the hazard function)} \\ &= \lim_{\Delta \downarrow 0} \frac{1}{\Delta} \cdot\frac{\Pr(\{t \le T < t + \Delta\} \cap \{T \ge t\})}{\Pr(T \ge t)} && \text{(definition of conditional probability)} \\ &= \lim_{\Delta \downarrow 0} \frac{1}{\Delta} \cdot\frac{\Pr(t \le T < t + \Delta)}{\Pr(T \ge t)} && \text{(subset property)} \\ &= \frac{1}{\Pr(T \ge t)} \cdot\lim_{\Delta \downarrow 0} \frac{\Pr(t \le T < t + \Delta)}{\Delta} && \text{(} \Pr(T \ge t) \text{ does not depend on } \Delta \text{)} \\ &= \frac{f(t)}{\Pr(T \ge t)} && \text{(density as a limit; } f \text{ is continuous at } t \text{)} \\ &= \frac{f(t)}{\operatorname{S}(t)} && \text{(} \Pr(T = t) = 0 \text{ for continuous } T \text{)} \end{aligned} \]
Proof. Let \(u < 0\). Every outcome has \(T(\omega) \ge 0 > u\), so \(\{T \ge u\} = \Omega\), which has probability \(1 > 0\), and \({\lambda}(u)\) is defined. For \(0 < \Delta \le -u\), the event \(\{u \le T < u + \Delta\}\) is empty, because \(u + \Delta \le 0\), so:
\[ \begin{aligned} {\lambda}(u) &= \lim_{\Delta \downarrow 0} \frac{\Pr(u \le T < u + \Delta \mid T \ge u)}{\Delta} && \text{(definition of the hazard function)} \\ &= \lim_{\Delta \downarrow 0} \frac{\Pr(\{u \le T < u + \Delta\} \cap \{T \ge u\})}{\Delta \cdot\Pr(T \ge u)} && \text{(definition of conditional probability)} \\ &= \lim_{\Delta \downarrow 0} \frac{\Pr(\emptyset \cap \Omega)}{\Delta \cdot\Pr(\Omega)} && \text{(the two events found above)} \\ &= \lim_{\Delta \downarrow 0} \frac{0}{\Delta \cdot 1} && \text{(} \emptyset \cap \Omega = \emptyset \text{; } \Pr(\emptyset) = 0 \text{; } \Pr(\Omega) = 1 \text{)} \\ &= 0 && \text{(simplify)} \end{aligned} \]
Then, for \(t \ge 0\):
\[ \begin{aligned} {\Lambda}(t) &= \int_{u=-\infty}^{t} {\lambda}(u)\,du && \text{(definition of the cumulative hazard)} \\ &= \int_{u=-\infty}^{0} {\lambda}(u)\,du + \int_{u=0}^{t} {\lambda}(u)\,du && \text{(split the integral at } 0 \text{)} \\ &= \int_{u=-\infty}^{0} 0\,du + \int_{u=0}^{t} {\lambda}(u)\,du && \text{(} {\lambda}(u) = 0 \text{ for } u < 0 \text{)} \\ &= \int_{u=0}^{t} {\lambda}(u)\,du && \text{(integrate)} \end{aligned} \]
Proof. Since \(\operatorname{S}(t) = 1 - F(t)\) (Theorem 14) and \(F\) has derivative \(f\) wherever \(f\) is continuous (Theorem 6), \(\operatorname{S}\) has derivative \(-f\) at those points. At each such \(u\) with \(\operatorname{S}(u) > 0\):
\[ \begin{aligned} \frac{d}{du}\mathopen{}\left(-\operatorname{log}\mathopen{}\left\{\operatorname{S}(u)\right\}\mathclose{}\right)\mathclose{} &= -\frac{1}{\operatorname{S}(u)} \cdot\frac{d}{du}\operatorname{S}(u) && \text{(chain rule)} \\ &= -\frac{1}{\operatorname{S}(u)} \cdot\mathopen{}\left(-f(u)\right)\mathclose{} && \text{(} \operatorname{S}' = -f \text{)} \\ &= \frac{f(u)}{\operatorname{S}(u)} && \text{(simplify)} \\ &= {\lambda}(u) && \text{(hazard equals density over survival)} \end{aligned} \]
The function \(-\operatorname{log}\mathopen{}\left\{\operatorname{S}(u)\right\}\mathclose{}\) is continuous, because \(\operatorname{S}\) is continuous for a continuous \(T\), and it has derivative \({\lambda}(u)\) at all but finitely many points, so it is an antiderivative of \({\lambda}\) for the fundamental theorem of calculus. As \(u \to -\infty\), \(\operatorname{S}(u) \to 1\), so \(-\operatorname{log}\mathopen{}\left\{\operatorname{S}(u)\right\}\mathclose{} \to 0\). Therefore:
\[ \begin{aligned} {\Lambda}(t) &\stackrel{\text{def}}{=}\int_{u=-\infty}^{t} {\lambda}(u)\,du && \text{(definition of the cumulative hazard)} \\ &= \mathopen{}\left[-\operatorname{log}\mathopen{}\left\{\operatorname{S}(u)\right\}\mathclose{}\right]\mathclose{}_{u=-\infty}^{t} && \text{(} -\log \operatorname{S}\text{ is an antiderivative of } {\lambda}\text{)} \\ &= -\operatorname{log}\mathopen{}\left\{\operatorname{S}(t)\right\}\mathclose{} - 0 && \text{(evaluate at the bounds)} \\ &= -\operatorname{log}\mathopen{}\left\{\operatorname{S}(t)\right\}\mathclose{} && \text{(simplify)} \end{aligned} \]
Exponentiating both sides of \(-{\Lambda}(t) = \operatorname{log}\mathopen{}\left\{\operatorname{S}(t)\right\}\mathclose{}\) gives Equation 1.
| Name | Symbols | Definition |
|---|---|---|
| Probability density function (PDF) | \(f(t), \operatorname{p}(t)\) | \(f \ge 0\) with \(\int_a^b f(u)\,du = \Pr(a \le T \le b)\) |
| Cumulative distribution function (CDF) | \(F(t)\) | \(\Pr(T\leq t)\) |
| Survival function | \(\operatorname{S}(t), \bar{F}(t)\) | \(\Pr(T > t)\) |
| Hazard function | \({\lambda}(t), \operatorname{h}(t)\) | \(\lim_{\Delta \downarrow 0} \Pr(t \le T < t + \Delta \mid T \ge t) / \Delta\) |
| Cumulative hazard function | \({\Lambda}(t), \operatorname{H}(t)\) | \(\int_{u=-\infty}^t {\lambda}(u)\,du\) |
| Log-hazard function | \(\eta(t)\) | \(\operatorname{log}\mathopen{}\left\{{\lambda}(t)\right\}\mathclose{}\) |