Last modified: 2026-11-07 04:59:05 (PST)
Definition 1 (Step) A step is the carrying out of one instruction.
Example 1 (Steps of a die roll) Carrying out the instruction “roll a six-sided die once” is one step. Carrying out the instruction “read the number facing up” is another step.
Definition 2 (Procedure) A procedure is a list of instructions that can be followed, one step at a time, until they are finished.
Definition 3 (Outcome) An outcome is a single, fully specified way that a procedure can end, recorded as one value.
Example 5 (Picking a car model from the Auto data) The Auto data record the number of engine cylinders and the fuel economy (miles per gallon) of 392 car models sold between 1970 and 1982. They come from the StatLib library at Carnegie Mellon University and ship with the book An Introduction to Statistical Learning and the R package ISLR2 (ISLR2::Auto). This site has a copy at data/auto.csv.
Pick one car model from the data without looking, and record its number of cylinders. Which model comes up is not determined in advance, so this procedure is a random experiment. Its possible outcomes are the cylinder counts that occur in the data:
Definition 5 (Sample space) The sample space of a random experiment, denoted \(\Omega\), is the set of all of its possible outcomes [2].
Example 6 (Sample space of a die roll) Rolling a six-sided die once has sample space \(\Omega= {\mathopen{}\left\{1, 2, 3, 4, 5, 6\right\}\mathclose{}}\).
Definition 6 (Event) An event is a subset of the sample space \(\Omega\) [3]: a set of outcomes.
Example 7 (Rolling an even number) In Example 6, the event “the roll is even” is \(A = {\mathopen{}\left\{2, 4, 6\right\}\mathclose{}}\).
Lemma 1 (The sample space is an event) The sample space \(\Omega\) is an event.
Proof. The sample space is a subset of itself, \(\Omega\subseteq \Omega\). When every subset of \(\Omega\) can be an event, that is enough. Otherwise the events are the subsets in a \(\sigma\)-algebra \(\mathcal{S}\) on \(\Omega\), and by its definition \(\mathcal{S}\) contains \(\Omega\) itself.
Example 8 (The die roll’s sample space as an event) In Example 6, every subset of \(\Omega= {\mathopen{}\left\{1, 2, 3, 4, 5, 6\right\}\mathclose{}}\) can be an event, including \(\Omega\) itself: the event “the roll is any of 1 through 6”, which always occurs.
Definition 7 (Occurrence of an event) An event \(A\) occurs when the outcome of the experiment is in \(A\), and does not occur when the outcome is not in \(A\).
Example 9 (Rolling a 4) In Example 7, if the die shows 4, the event “the roll is even”, \({\mathopen{}\left\{2, 4, 6\right\}\mathclose{}}\), occurs, because \(4 \in {\mathopen{}\left\{2, 4, 6\right\}\mathclose{}}\), and the event “the roll is at most 2”, \({\mathopen{}\left\{1, 2\right\}\mathclose{}}\), does not occur.
Definition 8 (Complement of an event) The complement of an event \(A\), denoted \(\neg A\), is the event that \(A\) does not occur: the set of elements \(\omega\) of the sample space \(\Omega\) that are not in \(A\), or equivalently, the set difference \({\Omega \setminus A}\):
\[ \begin{aligned} \neg A &\stackrel{\text{def}}{=}{{\mathopen{}\left\{\omega \in \Omega : \omega \notin A\right\}\mathclose{}}} \\ &= {\Omega \setminus A} \end{aligned} \]
Example 10 (Rolling an odd number) In Example 7, the complement of the event “the roll is even”, \(A = {\mathopen{}\left\{2, 4, 6\right\}\mathclose{}}\), is the event “the roll is odd”, \(\neg A = {\mathopen{}\left\{1, 3, 5\right\}\mathclose{}}\).
Definition 9 (Mutually exclusive events (disjoint, pairwise disjoint, mutually disjoint)) Finitely or countably many sets \(A_1, A_2, \ldots\), such as events, are mutually exclusive (also called disjoint, pairwise disjoint, or mutually disjoint) when no two of them share an element:
\[A_i \cap A_j = \emptyset \quad \text{for all } i \neq j\]
Example 11 (Low and high rolls) In Example 6, the events “the roll is at most 2”, \({\mathopen{}\left\{1, 2\right\}\mathclose{}}\), and “the roll is at least 5”, \({\mathopen{}\left\{5, 6\right\}\mathclose{}}\), are mutually exclusive: no roll is in both. The events “the roll is even”, \({\mathopen{}\left\{2, 4, 6\right\}\mathclose{}}\), and “the roll is at most 2”, \({\mathopen{}\left\{1, 2\right\}\mathclose{}}\), are not mutually exclusive: a roll of 2 is in both.
Definition 10 (Partition of an event) A partition of an event \(A\) is a finite or countably infinite collection of events \(A_1, A_2, \ldots\) that satisfies:
Remark 1 (A partition of the sample space). A partition of the sample space is the case \(A = \Omega\), which is allowed because \(\Omega\) is an event (Lemma 1).
Example 12 (Partitioning the die roll) In Example 6, the events “the roll is even”, \({\mathopen{}\left\{2, 4, 6\right\}\mathclose{}}\), and “the roll is odd”, \({\mathopen{}\left\{1, 3, 5\right\}\mathclose{}}\), are mutually exclusive, and their union is \(\Omega\), so they partition the sample space. So do the six single-outcome events \({\mathopen{}\left\{1\right\}\mathclose{}}, {\mathopen{}\left\{2\right\}\mathclose{}}, \ldots, {\mathopen{}\left\{6\right\}\mathclose{}}\). The events \({\mathopen{}\left\{1, 2\right\}\mathclose{}}\) and \({\mathopen{}\left\{5, 6\right\}\mathclose{}}\) from Example 11 do not partition \(\Omega\): their union leaves out 3 and 4.
Proposition 1 (An event and its complement partition the sample space) For any event \(A\), the events \(A\) and its complement \(\neg A\) are a partition of the sample space \(\Omega\).
Proof. The sample space \(\Omega\) is an event (Lemma 1), so a partition of it is defined. The complement \(\neg A\) is the set difference \({\Omega \setminus A}\), which is an event because the events are closed under complements in \(\Omega\), by the complement requirement of a \(\sigma\)-algebra. The events \(A\) and \(\neg A\) are mutually exclusive, because \(\neg A\) contains only elements not in \(A\). Their union is \(\Omega\), because each element of \(\Omega\) is either in \(A\) or not in \(A\).
Example 13 (Even and odd rolls) In Example 7 and Example 10, the event \(A = {\mathopen{}\left\{2, 4, 6\right\}\mathclose{}}\) and its complement \(\neg A = {\mathopen{}\left\{1, 3, 5\right\}\mathclose{}}\) are mutually exclusive, and their union is \(\Omega= {\mathopen{}\left\{1, 2, 3, 4, 5, 6\right\}\mathclose{}}\), so they partition \(\Omega\).
Definition 11 (Probability measure (probability function)) A probability measure (also called a probability function [4], Definition 1.2.4) on a sample space \(\Omega\), often denoted \(\Pr()\) or \(\mathbb{P}()\), is a measure on the events of \(\Omega\) that gives the whole sample space probability 1 [5]:
\[\Pr(\Omega) = 1\]
Example 14 (Probability measure for a fair die) For the die roll in Example 6, define \(\Pr(A) \stackrel{\text{def}}{=}\mathopen{}\left|A\right|\mathclose{} / 6\), where \(\mathopen{}\left|A\right|\mathclose{}\) is the number of outcomes in \(A\). This function is \(1/6\) times the counting measure \(\mu(A) = \mathopen{}\left|A\right|\mathclose{}\) on the die rolls (counting elements is a measure), so it is a measure on the events of \(\Omega\), and it gives
\[ \begin{aligned} \Pr(\Omega) &= 6/6 \\ &= 1. \end{aligned} \]
The event “the roll is even” from Example 7 has probability
\[ \begin{aligned} \Pr({\mathopen{}\left\{2, 4, 6\right\}\mathclose{}}) &= 3/6 \\ &= 1/2. \end{aligned} \]
Corollary 1 (Probability measures are finitely additive) Every probability measure is finitely additive: for any mutually exclusive events \(A_1, \ldots, A_n\),
\[\Pr(A_1 \cup \cdots \cup A_n) = \sum_{i=1}^n\Pr(A_i)\]
Proof. A probability measure is a measure (Definition 11), so measures are finitely additive applies to it.
Theorem 1 (Continuity of probability) If \(A_1 \supseteq A_2 \supseteq \cdots\) are events with \(\bigcap_{i=1}^{\infty} A_i = \emptyset\), then:
\[\lim_{n \to \infty} \Pr(A_n) = 0\]
Proof. For each \(i\), let \(B_i \stackrel{\text{def}}{=}A_i \setminus A_{i+1}\), the outcomes in \(A_i\) but not in \(A_{i+1}\). Each \(B_i = A_i \cap (\Omega\setminus A_{i+1})\) is an event, by closure properties of a \(\sigma\)-algebra.
The \(B_i\) are pairwise disjoint. For \(i < j\), \(B_j \subseteq A_j \subseteq A_{i+1}\), while \(B_i\) has no outcomes in \(A_{i+1}\), so \(B_i \cap B_j = \emptyset\).
\(A_n = \bigcup_{i=n}^{\infty} B_i\) for each \(n\). For \(i \ge n\), \(B_i \subseteq A_i \subseteq A_n\), so the union is contained in \(A_n\). Conversely, let \(\omega\in A_n\). Since \(\bigcap_{i} A_i = \emptyset\), \(\omega\) is not in every \(A_i\); since the \(A_i\) are nested, the indices \(i\) with \(\omega\in A_i\) are \(1, \ldots, m\) for some \(m \ge n\). Then \(\omega\in A_m\) and \(\omega\notin A_{m+1}\), so \(\omega\in B_m\).
\(\Pr(A_1)\) is finite. The events \(A_1\) and \(\Omega\setminus A_1\) are disjoint with union \(\Omega\), so Corollary 1 applies to them:
\[ \begin{aligned} \Pr(A_1) &\le \Pr(A_1) + \Pr(\Omega\setminus A_1) && \text{(} \Pr(\Omega\setminus A_1) \ge 0 \text{)} \\ &= \Pr(\Omega) && \text{(finite additivity of } \Pr \text{)} \\ &= 1 && \text{(definition of a probability measure)} \end{aligned} \]
The limit. For each \(n\):
\[ \begin{aligned} \Pr(A_n) &= {\Pr\mathopen{}\left(\bigcup_{i=n}^{\infty} B_i\right)\mathclose{}} && \text{(} A_n = \bigcup_{i=n}^{\infty} B_i \text{)} \\ &= \sum_{i=n}^{\infty} \Pr(B_i) && \text{(countable additivity; the } B_i \text{ are pairwise disjoint)} \end{aligned} \]
With \(n = 1\), this says the series \(\sum_{i=1}^{\infty} \Pr(B_i)\) has the finite total \(\Pr(A_1)\). So, for \(n \ge 2\):
\[ \begin{aligned} \Pr(A_n) &= \sum_{i=1}^{\infty} \Pr(B_i) - \sum_{i=1}^{n-1} \Pr(B_i) && \text{(split off the first } n - 1 \text{ terms; the total is finite)} \\ &= \Pr(A_1) - \sum_{i=1}^{n-1} \Pr(B_i) && \text{(the case } n = 1 \text{)} \end{aligned} \]
As \(n \to \infty\), the partial sums \(\sum_{i=1}^{n-1} \Pr(B_i)\) converge to the total \(\Pr(A_1)\), so \(\Pr(A_n) \to \Pr(A_1) - \Pr(A_1) = 0\).
Theorem 2 (Kolmogorov axioms) A function \(\Pr\) that assigns a real number \(\Pr(A)\) to each event \(A\) of a sample space \(\Omega\) is a probability measure if and only if it satisfies:
Proof. Suppose \(\Pr\) is a probability measure. It is a measure, so its values lie in the extended non-negative reals \([0, \infty]\), which gives axiom 1, and it is countably additive, which is axiom 3. Axiom 2 is the condition \(\Pr(\Omega) = 1\) in Definition 11.
Conversely, suppose \(\Pr\) satisfies the three axioms. By axiom 1, \(\Pr\) takes values in \([0, \infty]\), so axiom 3 says \(\Pr\) is countably additive in the sense of the definition of countable additivity. Then, by countable additivity and the empty set, \(\Pr(\emptyset)\) is 0 or \(\infty\); \(\Pr(\emptyset)\) is a real number, so \(\Pr(\emptyset) = 0\). So \(\Pr\) satisfies both conditions of the definition of a measure (\(\Pr(\emptyset) = 0\) and countable additivity) and is a measure on the events of \(\Omega\). With axiom 2, \(\Pr\) is a probability measure (Definition 11).
Theorem 3 (Probability of a subset’s intersection) If \(A\) and \(B\) are events and \(A\subseteq B\), then \(\Pr(A \cap B) = \Pr(A)\).
Proof. Since \(A \subseteq B\), every outcome in \(A\) is also in \(B\), so \(A \cap B = A\), and therefore \(\Pr(A \cap B) = \Pr(A)\).
Theorem 4 (An event and its complement sum to 1) For any event \(A\) and its complement \(\neg A\):
\[\Pr(A) + \Pr(\neg A) = 1\]
Proof. The events \(A\) and \(\neg A\) are disjoint, and their union is \(\Omega\).
\[ \begin{aligned} \Pr(A) + \Pr(\neg A) &= \Pr(A \cup \neg A) && \text{(additivity of probability for disjoint events)} \\ &= \Pr(\Omega) && \text{(} A \cup \neg A = \Omega\text{)} \\ &= 1 && \text{(probability of the sample space is 1)} \end{aligned} \]
Corollary 2 (Complement rule) For any event \(A\):
\[\Pr(\neg A) = 1 - \Pr(A)\]
Proof. Subtract \(\Pr(A)\) from both sides of Theorem 4.
Corollary 3 (Complement rule in probability (\(\pi\)) notation) If the probability of an event \(A\) is \(\Pr(A)=\pi\), then the probability that \(A\) does not occur is:
\[\Pr(\neg A)= 1 - \pi\]
Proof. \[ \begin{aligned} \Pr(\neg A) &= 1 - \Pr(A) && \text{(complement rule)} \\ &= 1 - \pi && \text{(substitute } \Pr(A) = \pi\text{)} \end{aligned} \]
Example 15 (Probability of not rolling a six) For a fair die, the event “the roll is a six” has probability \(\pi= 1/6\), so by Corollary 3 the probability of not rolling a six is \(1 - 1/6 = 5/6\).
Definition 12 (Choosing at random) Let \(S\) be a set with a measure \(\mu\) such that \(0 < \mu(S) < \infty\). To choose an element of \(S\) at random is to run a random experiment whose sample space is \(S\) and whose probability measure is the uniform distribution on \(S\).
Example 16 (Choosing at random from a finite set and from an interval) For a finite set \(S\), let \(\mu(A) = \mathopen{}\left|A\right|\mathclose{}\) be the counting measure. Then \(\Pr(A) = \mathopen{}\left|A\right|\mathclose{} / \mathopen{}\left|S\right|\mathclose{}\), so every element of \(S\) has the same probability \(1/\mathopen{}\left|S\right|\mathclose{}\).
For the interval \(S = [0, 1]\), let \(\mu(A)\) be the length of \(A\). Then \(\Pr(A)\) is the length of \(A\), so the probability that the chosen number lies in \([0.2, 0.5]\) is \(0.3\). The chosen number has the \(\text{Uniform}(0, 1)\) distribution.
The integers have infinite counting measure, so they have no choice at random in this sense.
Example 17 (Choosing a car at random from the Auto data) The Auto data of Example 5 have 392 car models. Let \(S\) be the set of car models in the data. Choosing a car at random from \(S\) (Definition 12), with the counting measure, gives each car probability \(1/\mathopen{}\left|S\right|\mathclose{}\), so the probability that the car has four cylinders is the fraction of cars with four cylinders:
# read the local copy of the Auto data (one row per car model):
auto <- read.csv("data/auto.csv")
# the event "four cylinders" is the set of rows where cylinders == 4;
# the mean of the logical vector is |A| / |S|, the probability in
# the definition of choosing at random:
mean(auto$cylinders == 4)
#> [1] 0.5076531Definition 13 (Odds) Let \(A\) be an event with probability \(\mathbb{P}(A) \in (0, 1)\). The odds of \(A\) are [7]
\[\omega\stackrel{\text{def}}{=}\frac{\mathbb{P}(A)}{1 - \mathbb{P}(A)} \in (0, \infty).\]
Example 18 (Odds that a car in the Auto data is American) In the Auto data of Example 5, let \(A\) be the event that a car chosen at random is American, which the data code as origin == 1. Its probability is the fraction of American cars, as in Example 17, and its odds follow from Definition 13:
The odds 1.67 say that an American car is about 1.7 times as likely as a car of non-American origin.
Definition 14 (Log-odds (also called the logit)) The log-odds, also called the logit, of an event \(A\) with probability \(\mathbb{P}(A) \in (0, 1)\) are the natural logarithm of its odds [8]:
\[\eta\stackrel{\text{def}}{=}{\operatorname{log}\mathopen{}\left\{\omega\right\}\mathclose{}} = {\operatorname{log}\mathopen{}\left\{\frac{\mathbb{P}(A)}{1 - \mathbb{P}(A)}\right\}\mathclose{}} \in \mathbb{R}.\]
Example 19 (Log-odds that a car in the Auto data is American) For the event \(A\) of Example 18, the log-odds of Definition 14 are:
The log-odds are positive because the odds exceed 1: an American car is more likely than not.
Definition 15 (Conditional probability) For two events \(A\) and \(B\) with \(\Pr(B) > 0\), the conditional probability of \(A\) given \(B\) [9], denoted \(\Pr(A \mid B)\), is:
\[\Pr(A \mid B) \stackrel{\text{def}}{=}\frac{\Pr(A \cap B)}{\Pr(B)}\]
Example 20 (Rolling a six, given an even roll) For a fair die (Example 14), let \(A = {\mathopen{}\left\{6\right\}\mathclose{}}\) and \(B = {\mathopen{}\left\{2, 4, 6\right\}\mathclose{}}\). Then \(A \cap B = {\mathopen{}\left\{6\right\}\mathclose{}}\), so:
\[ \begin{aligned} \Pr(A \mid B) &= \frac{\Pr(A \cap B)}{\Pr(B)} && \text{(definition of conditional probability)} \\ &= \frac{1/6}{3/6} && \text{(substitute the probabilities)} \\ &= \frac{1}{3} && \text{(simplify)} \end{aligned} \]
Theorem 5 (Law of conditional probability) For any two events \(A\) and \(B\) with \(\Pr(B) > 0\):
\[\Pr(A \cap B) = \Pr(A \mid B) \cdot\Pr(B)\]
Proof. Rearranging Definition 15:
\[ \begin{aligned} \Pr(A \mid B) &= \frac{\Pr(A \cap B)}{\Pr(B)} && \text{(definition of conditional probability)} \\ \Pr(A \cap B) &= \Pr(A \mid B) \cdot\Pr(B) && \text{(multiply both sides by } \Pr(B) \text{)} \end{aligned} \]
Example 21 (Applying the law of conditional probability) Suppose 30% of adults exercise regularly (\(\Pr(E) = 0.30\)), and among adults who exercise regularly, 60% have low blood pressure (\(\Pr(L \mid E) = 0.60\)).
Then, by Theorem 5, the probability that a randomly selected adult both exercises regularly and has low blood pressure is:
\[ \begin{aligned} \Pr(L \cap E) &= \Pr(L \mid E) \cdot\Pr(E) && \text{(law of conditional probability)} \\ &= 0.60 \cdot 0.30 && \text{(substitute the given values)} \\ &= 0.18 && \text{(multiply)} \end{aligned} \]
Theorem 6 (Law of total probability) If \(B_1, B_2, \ldots\) is a partition of the sample space, with \(\Pr(B_i) > 0\) for every \(i\), then for any event \(A\) [11]:
\[\Pr(A) = \sum_{i=1}^{\infty} \Pr(A \mid B_i) \cdot\Pr(B_i)\]
Proof. Since \(B_1, B_2, \ldots\) partition the sample space, the events \(A \cap B_1, A \cap B_2, \ldots\) are mutually exclusive and their union is \(A\). By countable additivity, and then by Theorem 5:
\[ \begin{aligned} \Pr(A) &= \sum_{i=1}^{\infty} \Pr(A \cap B_i) && \text{(countable additivity for partition of } A \text{)} \\&= \sum_{i=1}^{\infty} \Pr(A \mid B_i) \cdot\Pr(B_i) && \text{(law of conditional probability; } \Pr(B_i) > 0 \text{)} \end{aligned} \]
Theorem 7 (Bayes’ theorem) For any two events \(A\) and \(B\) with \(\Pr(A) > 0\) and \(\Pr(B) > 0\) [12]:
\[\Pr(A \mid B) = \frac{\Pr(B \mid A) \cdot\Pr(A)}{\Pr(B)}\]
Proof. By Definition 15 and Theorem 5:
\[ \begin{aligned} \Pr(A \mid B) &= \frac{\Pr(A \cap B)}{\Pr(B)} && \text{(definition of conditional probability)} \\ &= \frac{\Pr(B \cap A)}{\Pr(B)} && \text{(intersection is commutative: } A \cap B = B \cap A \text{)} \\ &= \frac{\Pr(B \mid A) \cdot\Pr(A)}{\Pr(B)} && \text{(law of conditional probability applied to } \Pr(B \cap A) \text{)} \end{aligned} \]
Example 22 (Positive predictive value of a medical test) Suppose a disease test has 99% sensitivity and 99% specificity, and the prevalence of the disease in the population is 7%.
Let \(D\) be the event “person has the disease” and \({+}\) be the event “test is positive”. Then:
\(\Pr({+}\mid D) = 0.99\) (sensitivity)
\(\Pr({-}\mid \neg D) = 0.99\) (specificity), so the false positive rate is
\[ \begin{aligned} \Pr({+}\mid \neg D) &= 1 - 0.99 \\ &= 0.01 \end{aligned} \]
\(\Pr(D) = 0.07\) (prevalence)
By Theorem 7, with the denominator expanded by the law of total probability over the partition \(\{D, \neg D\}\):
\[ \begin{aligned} \Pr(D \mid {+}) &= \frac{\Pr({+}\mid D) \cdot\Pr(D)}{\Pr({+})} && \text{(Bayes' theorem)} \\ &= \frac{\Pr({+}\mid D) \cdot\Pr(D)}{\Pr({+}\mid D) \cdot\Pr(D) + \Pr({+}\mid \neg D) \cdot\Pr(\neg D)} && \text{(law of total probability)} \\ &= \frac{0.99 \cdot 0.07}{0.99 \cdot 0.07 + 0.01 \cdot 0.93} && \text{(substitute the given values)} \\ &= \frac{0.0693}{0.0693 + 0.0093} && \text{(multiply each term in the numerator and denominator)} \\ &= \frac{0.0693}{0.0786} && \text{(add the denominator's two terms)} \\ &\approx 0.88 && \text{(divide)} \end{aligned} \]
Even with a highly accurate test (99% sensitive and 99% specific), only about 88% of people who test positive actually have the disease, because the disease prevalence is relatively low (7%).
Exercise 1 (Turn a conditional around) Definition 15 defines \(\Pr(A \mid B)\) in terms of \(\Pr(A \cap B)\). Write the same definition with \(A\) and \(B\) swapped, then eliminate \(\Pr(A \cap B)\) between the two.
Your result should express \(\Pr(A \mid B)\) using \(\Pr(B \mid A)\), \(\Pr(A)\) and \(\Pr(B)\), and none of them jointly.
Solution 1. Multiplying Definition 15 through by its denominator, in each direction,
\[\Pr(A \cap B) = \Pr(A \mid B)\,\Pr(B), \qquad \Pr(A \cap B) = \Pr(B \mid A)\,\Pr(A)\]
The left-hand sides are the same quantity, so the right-hand sides are equal. Dividing by \(\Pr(B)\) gives Bayes’ rule (Theorem 7):
\[\Pr(A \mid B) = \frac{\Pr(B \mid A)\,\Pr(A)}{\Pr(B)} \tag{1}\]
Nothing was assumed beyond \(\Pr(A) > 0\) and \(\Pr(B) > 0\), which the two conditionals need in order to be defined at all.
As Hutchinson emphasizes [13], each component carries standard Bayesian terminology:
From this formula, the direction of effects is transparent: increasing the prior \(\Pr(A)\) or the likelihood \(\Pr(B \mid A)\) increases the posterior \(\Pr(A \mid B)\), while increasing the probability of the evidence \(\Pr(B)\) decreases it.
Exercise 2 (A test that is right 99% of the time) A screening test for a disease is positive for \(99\%\) of people who have it and negative for \(99\%\) of people who do not. One person in a thousand has the disease.
A randomly screened person tests positive. What is the probability that they have the disease? Guess first, then compute it with Equation 1.
Solution 2. Write \(S\) for having the disease and \({+}\) for a positive test. The three given numbers are
\[\Pr({+}\mid S) = 0.99, \qquad \Pr({+}\mid \text{not } S) = 0.01, \qquad \Pr(S) = 0.001\]
Equation 1 needs \(\Pr({+})\), the overall chance of a positive result. Assemble it with Theorem 6, splitting on whether the person is sick:
\[\begin{aligned} \Pr({+}) &= \Pr({+}\mid S)\Pr(S) + \Pr({+}\mid \text{not } S)\Pr(\text{not } S) \\ &= (0.99)(0.001) + (0.01)(0.999) \\ &= 0.00099 + 0.00999 \\ &= 0.01098 \end{aligned}\]
Then
\[ \begin{aligned} \Pr(S \mid {+}) &= \frac{\Pr({+}\mid S)\,\Pr(S)}{\Pr({+})} \\ &= \frac{0.00099}{0.01098} \\ &\approx 0.0902 \end{aligned} \]
About 9%, not the \(99\%\) most people guess.
The reason is the base rate [14]. Out of \(100{,}000\) people screened, about \(100\) have the disease and \(99\) of them test positive, while \(99{,}900\) do not have it and about \(999\) of them test positive anyway. The false positives outnumber the true positives ten to one, because there are a thousand times more people available to produce them.
The base rate is fundamental to evaluating screening tests and binary classifiers, and it illustrates why raw accuracy can be misleading on rare events.
viewof brSens = Inputs.range([0.5, 1], {value: 0.99, step: 0.001, label: "P(+ | S), sensitivity"})
viewof brSpec = Inputs.range([0.5, 1], {value: 0.99, step: 0.001, label: "P(\u2212 | not S), specificity"})
viewof brPrev = Inputs.range([0.0001, 0.5], {value: 0.001, transform: Math.log, format: d3.format(".4~f"), label: "P(S), prevalence"}){
const n = (v) => Math.round(v).toLocaleString("en-US");
return md`Out of ${n(brN)} people screened, ${n(brSick)} have the disease and ${n(brTP)} of them test positive;
of the ${n(brN - brSick)} who do not, ${n(brFP)} test positive anyway.
So of the ${n(brTP + brFP)} positives, ${n(brTP)} are sick:
P(S | +) = **${(100 * brPost(brPrev)).toFixed(1)}%**.`;
}Plot.plot({
ariaLabel: 'Of 100,000 people screened, ' +
'the ones who test positive, ' +
'as one bar split into the sick, ' +
'in orange, ' +
'and the healthy, ' +
'in blue.',
width: 380, height: 120, marginLeft: 10, marginRight: 20,
x: {label: "people who test positive, out of 100,000 screened"},
color: {domain: ["sick", "not sick"], range: ["#ff7f0e", "#1f77b4"], legend: true},
marks: [
Plot.barX([{who: "sick", n: brTP}, {who: "not sick", n: brFP}],
{x: "n", fill: "who", insetTop: 10, insetBottom: 10}),
Plot.ruleX([0])
]
})Plot.plot({
ariaLabel: 'The chance of disease after a positive test, ' +
'plotted against the prevalence on a log scale, ' +
'for the chosen sensitivity and specificity, ' +
'with the chosen prevalence marked.',
width: 380, height: 230, grid: true,
x: {type: "log", domain: [0.0001, 0.5], label: "P(S), prevalence (log scale)",
ticks: [0.0001, 0.001, 0.01, 0.1, 0.5], tickFormat: d3.format(".2~%")},
y: {domain: [0, 1], label: "P(S | +)"},
marks: [
Plot.line(d3.range(-4, Math.log10(0.5) + 0.001, 0.02).map((e) => 10 ** e),
{x: (p) => p, y: brPost, stroke: "#555"}),
Plot.dot([brPrev], {x: (p) => p, y: brPost, r: 5, fill: "#ff7f0e"})
]
})Video lecture
Hutchinson’s Probability Refresher (27 min) covers conditional distributions, the law of total probability, the chain rule of probability, and Bayes’ rule [15]. The login for the video site is posted on Canvas.
Another explanation
The University of Washington’s CSE 312 materials and videos by Alex Tsun teach probability and statistics for computing, and cover counting, conditional probability and Bayes’ rule from a computer science viewpoint.
5MinuteAI’s five-minute videos follow the same course [16]: introduction to discrete probability and conditional probability.
Another explanation
Lange’s textbook [17] reviews probability and expectation in Section 1.2 and conditional probability in Section 1.3.