Last modified: 2026-10-27 23:24:14 (PDT)
Classification is a core problem in statistics and machine learning: we seek to assign individuals or observations to one of several discrete categories based on available data. In medicine and epidemiology, classification problems arise constantly; for example, a clinician decides whether a patient has a disease based on evidence such as:
A test can look highly accurate in isolation, yet its predictive value for an individual patient depends heavily on the prevalence of the condition in the population being tested. Understanding this interplay requires Bayes’ theorem and the law of total probability [1].
Definition 1 (Classification) A classification problem is a statistical problem in which we seek to assign each observation to one of two or more discrete categories based on its observed features or predictors \(x\).
Source: Wikipedia contributors [2].
Definition 2 (Class) The discrete categories of a classification problem (Definition 1) are its classes. Let \(\mathcal{C}\) be a set of \(K\) classes (for example, \(\mathcal{C}= {\mathopen{}\left\{1, \dots, K\right\}\mathclose{}}\), or a set of text labels).
Definition 3 (Classification rule) A classification rule \(c\) maps features to classes (Definition 2):
\[c: x \mapsto c(x) \in \mathcal{C} \tag{1}\]
Definition 4 (Binary classification) A binary classification problem is a classification problem whose set of classes \(\mathcal{C}\) consists of \(K = 2\) distinct categories, one representing the presence of a condition of interest (\(k_1\)) and the other representing its absence (\(k_0\)):
\[\mathcal{C}= {\mathopen{}\left\{k_1, k_0\right\}\mathclose{}}, \quad \mathopen{}\left|\mathcal{C}\right|\mathclose{} = 2 \tag{2}\]
Source: Wikipedia contributors [2].
Definition 5 (Positive classification) In a binary classification problem with \(\mathcal{C}= {\mathopen{}\left\{k_1, k_0\right\}\mathclose{}}\), for an observation drawn from a population and assigned class \(Y \in \mathcal{C}\) (for example, \(Y = c(X)\) when a classification rule \(c\) classifies the observation from its feature \(X\)), a positive classification (also called positive, and written \({+}\)) is the event that the observation is assigned to \(k_1\):
\[{+}\stackrel{\text{def}}{=}{\mathopen{}\left\{Y = k_1\right\}\mathclose{}} \tag{3}\]
Source: Wikipedia contributors [2]; Wikipedia contributors [3].
Definition 6 (Negative classification) In a binary classification problem with \(\mathcal{C}= {\mathopen{}\left\{k_1, k_0\right\}\mathclose{}}\), for an observation drawn from a population and assigned class \(Y \in \mathcal{C}\) (for example, \(Y = c(X)\) when a classification rule \(c\) classifies the observation from its feature \(X\)), a negative classification (also called negative, and written \({-}\)) is the event that the observation is assigned to \(k_0\):
\[{-}\stackrel{\text{def}}{=}{\mathopen{}\left\{Y = k_0\right\}\mathclose{}} \tag{4}\]
Source: Wikipedia contributors [2]; Wikipedia contributors [3].
Exercise 1 (Negative classification as a complement) In a binary classification problem with \(\mathcal{C}= {\mathopen{}\left\{k_1, k_0\right\}\mathclose{}}\), show that when each observation is assigned a class \(Y \in \mathcal{C}\), the negative classification event \({-}\) is the complement of the positive classification event \({+}\): \({-}= \neg {+}\).
Solution 1. Because \(\mathcal{C}= {\mathopen{}\left\{k_1, k_0\right\}\mathclose{}}\) with \(\mathopen{}\left|\mathcal{C}\right|\mathclose{} = 2\) (Definition 4), the classes \(k_1\) and \(k_0\) are distinct, so \(Y \in \mathcal{C}\) equals \(k_0\) if and only if \(Y \neq k_1\):
\[ \begin{aligned} {-} &= {\mathopen{}\left\{Y = k_0\right\}\mathclose{}} && \text{(definition of negative classification)} \\ &= {\mathopen{}\left\{Y \neq k_1\right\}\mathclose{}} && \text{(since } Y \in {\mathopen{}\left\{k_1, k_0\right\}\mathclose{}} \text{ with } k_1 \neq k_0\text{)} \\ &= \neg {\mathopen{}\left\{Y = k_1\right\}\mathclose{}} && \text{(definition of complement)} \\ &= \neg {+} && \text{(definition of positive classification)} \end{aligned} \tag{5}\]
Theorem 1 (Negative classification is the complement of positive classification) In a binary classification problem with \(\mathcal{C}= {\mathopen{}\left\{k_1, k_0\right\}\mathclose{}}\), when each observation is assigned a class \(Y \in \mathcal{C}\), the negative classification event \({-}\) is the complement of the positive classification event \({+}\):
\[{-}= \neg {+} \tag{6}\]
Proof. This is the solution to Exercise 1, worked in Solution 1.
Example 1 (A diagnostic test as a classification rule) In a binary classification problem, a COVID-19 test assigns each person tested to one of \(K = 2\) classes, “has COVID-19” (\(k_1\)) or “does not have COVID-19” (\(k_0\)), based on a single feature \(x\): whether the test’s result is “positive” or “negative”. The test is the classification rule that assigns a positive result to \(k_1\) and a negative result to \(k_0\):
\[ c(x) \stackrel{\text{def}}{=} \begin{cases} \text{has COVID-19} & \text{if } x = \text{positive} \\ \text{does not have COVID-19} & \text{if } x = \text{negative} \end{cases} \tag{7}\]
For a person drawn from a population with test result \(X\), the event that the test is positive is \({+}= {\mathopen{}\left\{c(X) = \text{has COVID-19}\right\}\mathclose{}}\), and the event that the test is negative is \({-}= {\mathopen{}\left\{c(X) = \text{does not have COVID-19}\right\}\mathclose{}}\).
Remark 1 (Labels in binary classification). In a binary classification problem, the two classes \(k_1\) and \(k_0\) are often labeled “positive” and “negative”, or, in medical diagnostic testing, “has the disease” and “does not have the disease”.
Definition 7 (Sensitivity) The sensitivity of a diagnostic test is the probability that the test is positive (\({+}\)), given that the person tested has the disease (\(D\)):
\[\text{sensitivity} \stackrel{\text{def}}{=}\Pr({+}\mid D) \tag{8}\]
Source: Wikipedia contributors [3].
Example 2 (Sensitivity of a COVID-19 test) Suppose a COVID-19 test is positive for 99% of the people who have COVID-19. Let \(D\) be the event “the person has COVID-19” and \({+}\) the event “the test is positive”. Then the test’s sensitivity (Definition 7) is:
\[\Pr({+}\mid D) = 0.99 \tag{9}\]
Definition 8 (Specificity) The specificity of a diagnostic test is the probability that the test is negative (\({-}\)), given that the person tested does not have the disease (\(\neg D\)):
\[\text{specificity} \stackrel{\text{def}}{=}\Pr({-}\mid \neg D) \tag{10}\]
Source: Wikipedia contributors [3].
Example 3 (Specificity of a COVID-19 test) Suppose the COVID-19 test of Example 2 is negative for 99% of the people who do not have COVID-19. Then its specificity (Definition 8) is:
\[\Pr({-}\mid \neg D) = 0.99 \tag{11}\]
By the complement rule and Theorem 1, its false positive rate is:
\[ \begin{aligned} \Pr({+}\mid \neg D) &= 1 - \Pr(\neg {+}\mid \neg D) && \text{(complement rule, conditional on } \neg D \text{)} \\ &= 1 - \Pr({-}\mid \neg D) && \text{(negative classification is the complement of positive)} \\ &= 1 - 0.99 && \text{(substitute the specificity)} \\ &= 0.01 && \text{(subtract)} \end{aligned} \tag{12}\]
Definition 9 (Prevalence) The prevalence of a disease in a population is the probability that a person drawn from that population has the disease (\(D\)):
\[\text{prevalence} \stackrel{\text{def}}{=}\Pr(D) \tag{13}\]
Source: Wikipedia contributors [4].
Example 4 (Prevalence of COVID-19) Suppose 7% of the population being tested with the COVID-19 test of Example 2 has COVID-19. Then the prevalence (Definition 9) is:
\[\Pr(D) = 0.07 \tag{14}\]
and, by the complement rule, the probability that a person drawn from that population does not have COVID-19 is:
\[ \begin{aligned} \Pr(\neg D) &= 1 - \Pr(D) && \text{(complement rule)} \\ &= 1 - 0.07 && \text{(substitute the prevalence)} \\ &= 0.93 && \text{(subtract)} \end{aligned} \tag{15}\]
Definition 10 (Positive predictive value (PPV)) The positive predictive value of a diagnostic test is the probability that the person tested has the disease (\(D\)), given that the test is positive (\({+}\)):
\[\text{PPV} \stackrel{\text{def}}{=}\Pr(D \mid {+}) \tag{16}\]
Source: Wikipedia contributors [5].
Example 5 (PPV of a COVID-19 test) For the COVID-19 test of Example 2 and Example 3, used in the population of Example 4, Bayes’ theorem gives the positive predictive value (Definition 10):
\[\Pr(D \mid {+}) \approx 0.88 \tag{17}\]
Even with a highly accurate test (99% sensitive and 99% specific), only about 88% of the people who test positive have COVID-19, because the prevalence (7%) is low enough that false positives make up a meaningful fraction of all positive tests.
Definition 11 (Negative predictive value (NPV)) The negative predictive value of a diagnostic test is the probability that the person tested does not have the disease (\(\neg D\)), given that the test is negative (\({-}\)):
\[\text{NPV} \stackrel{\text{def}}{=}\Pr(\neg D \mid {-}) \tag{18}\]
Source: Wikipedia contributors [5].
Example 6 (NPV of a COVID-19 test) For the COVID-19 test of Example 2 and Example 3, used in the population of Example 4, the probability of a negative test for a person with COVID-19 is, by the complement rule and Theorem 1:
\[ \begin{aligned} \Pr({-}\mid D) &= \Pr(\neg {+}\mid D) && \text{(negative classification is the complement of positive)} \\ &= 1 - \Pr({+}\mid D) && \text{(complement rule, conditional on } D \text{)} \\ &= 1 - 0.99 && \text{(substitute the sensitivity)} \\ &= 0.01 && \text{(subtract)} \end{aligned} \tag{19}\]
By Bayes’ theorem, with the denominator expanded by the law of total probability over the partition \({\mathopen{}\left\{D, \neg D\right\}\mathclose{}}\), the negative predictive value (Definition 11) is:
\[ \begin{aligned} \Pr(\neg D \mid {-}) &= \frac{\Pr({-}\mid \neg D) \cdot\Pr(\neg D)}{\Pr({-})} && \text{(Bayes' theorem)} \\ &= \frac{\Pr({-}\mid \neg D) \cdot\Pr(\neg D)}{\Pr({-}\mid \neg D) \cdot\Pr(\neg D) + \Pr({-}\mid D) \cdot\Pr(D)} && \text{(law of total probability)} \\ &= \frac{0.99 \cdot 0.93}{0.99 \cdot 0.93 + 0.01 \cdot 0.07} && \text{(substitute the given values)} \\ &= \frac{0.9207}{0.9207 + 0.0007} && \text{(multiply each term in the numerator and denominator)} \\ &= \frac{0.9207}{0.9214} && \text{(add the denominator's two terms)} \\ &\approx 0.9992 && \text{(divide)} \end{aligned} \tag{20}\]
A negative result from this test is very reliable: only about 8 in 10,000 people who test negative have COVID-19.
Exercise 2 (Probability of a positive test) Derive an expression for the probability \(\Pr({+})\) that a diagnostic test is positive in terms of:
Solution 2. By the law of total probability over the partition \({\mathopen{}\left\{D, \neg D\right\}\mathclose{}}\), the complement rule, and Theorem 1:
\[ \begin{aligned} \Pr({+}) &= \Pr({+}\mid D) \cdot\Pr(D) + \Pr({+}\mid \neg D) \cdot\Pr(\neg D) && \text{(law of total probability)} \\ &= \text{sensitivity} \cdot\Pr(D) + \Pr({+}\mid \neg D) \cdot\Pr(\neg D) && \text{(definition of sensitivity)} \\ &= \text{sensitivity} \cdot\text{prevalence} + \Pr({+}\mid \neg D) \cdot\Pr(\neg D) && \text{(definition of prevalence)} \\ &= \text{sensitivity} \cdot\text{prevalence} + \mathopen{}\left(1 - \Pr(\neg {+}\mid \neg D)\right)\mathclose{} \cdot\Pr(\neg D) && \text{(complement rule, conditional on } \neg D \text{)} \\ &= \text{sensitivity} \cdot\text{prevalence} + \mathopen{}\left(1 - \Pr({-}\mid \neg D)\right)\mathclose{} \cdot\Pr(\neg D) && \text{(negative classification is the complement of positive)} \\ &= \text{sensitivity} \cdot\text{prevalence} + \mathopen{}\left(1 - \text{specificity}\right)\mathclose{} \cdot\Pr(\neg D) && \text{(definition of specificity)} \\ &= \text{sensitivity} \cdot\text{prevalence} + \mathopen{}\left(1 - \text{specificity}\right)\mathclose{} \cdot\mathopen{}\left(1 - \Pr(D)\right)\mathclose{} && \text{(complement rule)} \\ &= \text{sensitivity} \cdot\text{prevalence} + \mathopen{}\left(1 - \text{specificity}\right)\mathclose{} \cdot\mathopen{}\left(1 - \text{prevalence}\right)\mathclose{} && \text{(definition of prevalence)} \end{aligned} \tag{21}\]
Theorem 2 (Probability of a positive test) Let \(\text{sens}\) be the sensitivity of a diagnostic test, \(\text{spec}\) its specificity, and \(\pi\) the prevalence of the disease. Then the probability \(\Pr({+})\) that the test is positive is:
\[ \Pr({+}) = \text{sens} \cdot\pi + \mathopen{}\left(1 - \text{spec}\right)\mathclose{} \cdot\mathopen{}\left(1 - \pi\right)\mathclose{} \tag{22}\]
Proof. This is the solution to Exercise 2, worked in Solution 2.
Exercise 3 (PPV from Bayes’ theorem) Derive an expression for the positive predictive value of a diagnostic test in terms of:
Solution 3. Abbreviate the sensitivity as \(\text{sens}\), the specificity as \(\text{spec}\), and the prevalence as \(\pi\). By Bayes’ theorem, with the probability of a positive test from Exercise 2:
\[ \begin{aligned} \text{PPV} &\stackrel{\text{def}}{=}\Pr(D \mid {+}) && \text{(definition of PPV)} \\ &= \frac{\Pr({+}\mid D) \cdot\Pr(D)}{\Pr({+})} && \text{(Bayes' theorem)} \\ &= \frac{\text{sens} \cdot\Pr(D)}{\Pr({+})} && \text{(definition of sensitivity)} \\ &= \frac{\text{sens} \cdot\pi}{\Pr({+})} && \text{(definition of prevalence)} \\ &= \frac{\text{sens} \cdot\pi}{\text{sens} \cdot\pi+ \mathopen{}\left(1 - \text{spec}\right)\mathclose{} \cdot\mathopen{}\left(1 - \pi\right)\mathclose{}} && \text{(probability of a positive test)} \end{aligned} \tag{23}\]
Theorem 3 (PPV from Bayes’ theorem) Let \(\text{sens}\) be the sensitivity of a diagnostic test, \(\text{spec}\) its specificity, and \(\pi\) the prevalence of the disease. If the probability of a positive test is positive, \(\text{sens} \cdot\pi+ \mathopen{}\left(1 - \text{spec}\right)\mathclose{} \cdot\mathopen{}\left(1 - \pi\right)\mathclose{} > 0\), then the positive predictive value of the test is:
\[ \text{PPV} = \frac{\text{sens} \cdot\pi} {\text{sens} \cdot\pi+ \mathopen{}\left(1 - \text{spec}\right)\mathclose{} \cdot\mathopen{}\left(1 - \pi\right)\mathclose{}} \tag{24}\]
Source: Wikipedia contributors [1].
Proof. This is the solution to Exercise 3, worked in Solution 3, which builds on Theorem 2.
Exercise 4 (PPV in terms of two ratios) Rewrite the expression for the positive predictive value from Theorem 3 so that the sensitivity, specificity, and prevalence appear only through the ratio \(\frac{1 - \text{spec}}{\text{sens}}\) of the false positive rate to the sensitivity and the ratio \(\frac{1 - \pi}{\pi}\) of non-diseased to diseased people in the population.
Solution 4. Write \(a \stackrel{\text{def}}{=}\text{sens} \cdot\pi\) and \(b \stackrel{\text{def}}{=}\mathopen{}\left(1 - \text{spec}\right)\mathclose{} \cdot\mathopen{}\left(1 - \pi\right)\mathclose{}\), so that Theorem 3 reads \(\text{PPV} = \frac{a}{a + b}\), with \(a > 0\) whenever the sensitivity and prevalence are positive. Then:
\[ \begin{aligned} \text{PPV} &= \frac{a}{a + b} && \text{(PPV from Bayes' theorem)} \\ &= \frac{1}{\mathopen{}\left(\frac{a + b}{a}\right)\mathclose{}} && \text{(a fraction equals one over its reciprocal, since } a > 0 \text{)} \\ &= \frac{1}{\frac{a}{a} + \frac{b}{a}} && \text{(distribute the division over the sum in the numerator)} \\ &= \frac{1}{1 + \frac{b}{a}} && \text{(} \tfrac{a}{a} = 1 \text{)} \\ &= \frac{1}{1 + \frac{\mathopen{}\left(1 - \text{spec}\right)\mathclose{} \cdot\mathopen{}\left(1 - \pi\right)\mathclose{}}{\text{sens} \cdot\pi}} && \text{(substitute the definitions of } a \text{ and } b \text{)} \\ &= \frac{1}{1 + \frac{1 - \text{spec}}{\text{sens}} \cdot\frac{1 - \pi}{\pi}} && \text{(a quotient of products is the product of the quotients)} \end{aligned} \tag{25}\]
Theorem 4 (PPV in terms of sensitivity, specificity, and prevalence) The positive predictive value of a diagnostic test with positive sensitivity, used in a population with positive prevalence, depends on its sensitivity, its specificity, and the prevalence of the disease only through two ratios:
Specifically:
\[ \text{PPV} = \frac{1}{1 + \frac{1 - \text{spec}}{\text{sens}} \cdot\frac{1 - \pi}{\pi}} \tag{26}\]
Proof. This result is the solution to Exercise 4, worked in Solution 4, which builds on Theorem 3 and Theorem 2.
Example 7 (PPV of a COVID-19 test, from the two ratios) For the COVID-19 test of Example 2 and Example 3, used in the population of Example 4, Theorem 4 gives:
\[ \begin{aligned} \text{PPV} &= \frac{1}{1 + \frac{1 - 0.99}{0.99} \cdot\frac{1 - 0.07}{0.07}} && \text{(PPV in terms of the two ratios, with the given values)} \\ &= \frac{1}{1 + \frac{0.01}{0.99} \cdot\frac{0.93}{0.07}} && \text{(subtract in each numerator)} \\ &\approx \frac{1}{1 + 0.0101 \cdot\frac{0.93}{0.07}} && \text{(divide $0.01 / 0.99$)} \\ &\approx \frac{1}{1 + 0.0101 \cdot 13.29} && \text{(divide $0.93 / 0.07$)} \\ &\approx \frac{1}{1 + 0.134} && \text{(multiply)} \\ &= \frac{1}{1.134} && \text{(add)} \\ &\approx 0.88 && \text{(divide)} \end{aligned} \tag{27}\]
This value matches Example 5.
If the same test were used in a population with prevalence 0.1% instead, and its sensitivity and specificity were still 99% in that population, the ratio of non-diseased to diseased people would be \(0.999 / 0.001 = 999\), and the PPV would fall to
\[ \begin{aligned} \text{PPV} &= \frac{1}{1 + \frac{0.01}{0.99} \cdot 999} && \text{(PPV in terms of the two ratios, with the new prevalence)} \\ &\approx \frac{1}{1 + 0.0101 \cdot 999} && \text{(divide)} \\ &\approx \frac{1}{1 + 10.09} && \text{(multiply)} \\ &= \frac{1}{11.09} && \text{(add)} \\ &\approx 0.09 && \text{(divide)} \end{aligned} \tag{28}\]
so about 9 in 10 positive results would be false positives.
Remark 2 (The PPV depends on the population, not only on the test). The sensitivity and specificity describe the test’s performance in a specified setting, and may differ across settings (for example, with the case mix of the people tested or the threshold used to call a result positive), but the ratio \(\frac{1 - \pi}{\pi}\) in Theorem 4 describes the population being tested. When the disease is rare, that ratio is large, so even a small false positive rate can make the PPV low, as Example 7 shows.
So far, we have compared a test with the true disease status. The same tools apply whenever we compare two classifiers that label the same observations: for example, a new diagnostic test against a reference test, or an automated grader (an “agent”) against a human grader. We call the classifier being evaluated the agent and the one we compare it with the reference.
Definition 12 (Agreement table) For \(n\) observations that an agent and a reference each classify as positive or negative, the agreement table counts the four combinations:
| Reference positive | Reference negative | Total | |
|---|---|---|---|
| Agent positive | \(a\) | \(b\) | \(a + b\) |
| Agent negative | \(c\) | \(d\) | \(c + d\) |
| Total | \(a + c\) | \(b + d\) | \(n\) |
The cells \(a\) and \(d\) count agreements, the cells \(b\) and \(c\) count disagreements, and the four cells together count every observation:
\[n \stackrel{\text{def}}{=}a + b + c + d \tag{29}\]
Source: Wikipedia contributors [6].
Example 8 (Agreement table for a COVID-19 test) Apply the COVID-19 test of Example 2 and Example 3 to \(n = 10{,}000\) people from the population of Example 4, with the true disease status as the reference. With 7% prevalence, \(0.07 \times 10{,}000 = 700\) people have COVID-19 and \(10{,}000 - 700 = 9{,}300\) do not. With 99% sensitivity, the test is positive for \(0.99 \times 700 = 693\) of the 700; with 99% specificity, it is negative for \(0.99 \times 9{,}300 = 9{,}207\) of the 9,300. So the test is positive for \(9{,}300 - 9{,}207 = 93\) people without COVID-19 and negative for \(700 - 693 = 7\) people with it. The agreement table (Definition 12) is:
| COVID-19 | No COVID-19 | Total | |
|---|---|---|---|
| Test positive | \(a = 693\) | \(b = 93\) | \(786\) |
| Test negative | \(c = 7\) | \(d = 9{,}207\) | \(9{,}214\) |
| Total | \(700\) | \(9{,}300\) | \(10{,}000\) |
Definition 13 (Overall agreement) Let:
The overall agreement of the agent and the reference on that population is the probability that they assign the same class to the observation:
\[p_o \stackrel{\text{def}}{=}{\Pr\mathopen{}\left(A \cap R\right)\mathclose{}} + {\Pr\mathopen{}\left(\neg A \cap \neg R\right)\mathclose{}} \tag{30}\]
Source: Cohen [7].
Example 9 (Overall agreement for the COVID-19 test) Apply the COVID-19 test of Example 2 and Example 3 to the population of Example 4, with the test as the agent (\(A = {+}\)) and the disease status as the reference (\(R = D\)). By the multiplication rule for conditional probability, with sensitivity \({\Pr\mathopen{}\left({+}\mid D\right)\mathclose{}} = 0.99\), specificity \({\Pr\mathopen{}\left({-}\mid \neg D\right)\mathclose{}} = 0.99\), prevalence \({\Pr\mathopen{}\left(D\right)\mathclose{}} = 0.07\), and \({\Pr\mathopen{}\left(\neg D\right)\mathclose{}} = 0.93\) (Example 4), the probability that a person has COVID-19 and tests positive is:
\[ \begin{aligned} {\Pr\mathopen{}\left({+}\cap D\right)\mathclose{}} &= {\Pr\mathopen{}\left({+}\mid D\right)\mathclose{}} \cdot{\Pr\mathopen{}\left(D\right)\mathclose{}} && \text{(multiplication rule for conditional probability)} \\ &= 0.99 \cdot 0.07 && \text{(substitute the sensitivity and prevalence)} \\ &= 0.0693 && \text{(multiply)} \end{aligned} \tag{31}\]
and the probability that a person does not have COVID-19 and tests negative is:
\[ \begin{aligned} {\Pr\mathopen{}\left({-}\cap \neg D\right)\mathclose{}} &= {\Pr\mathopen{}\left({-}\mid \neg D\right)\mathclose{}} \cdot{\Pr\mathopen{}\left(\neg D\right)\mathclose{}} && \text{(multiplication rule for conditional probability)} \\ &= 0.99 \cdot 0.93 && \text{(substitute the specificity and } {\Pr\mathopen{}\left(\neg D\right)\mathclose{}} \text{)} \\ &= 0.9207 && \text{(multiply)} \end{aligned} \tag{32}\]
Then the overall agreement is:
\[ \begin{aligned} p_o &= {\Pr\mathopen{}\left({+}\cap D\right)\mathclose{}} + {\Pr\mathopen{}\left({-}\cap \neg D\right)\mathclose{}} && \text{(definition of } p_o \text{)} \\ &= 0.0693 + 0.9207 && \text{(substitute the two joint probabilities)} \\ &= 0.99 && \text{(add)} \end{aligned} \tag{33}\]
Definition 14 (Estimated overall agreement) The estimated overall agreement of an agreement table is the sample proportion of observations on which the agent and the reference assign the same class, which estimates the overall agreement \(p_o\):
\[\widehat{p}_o \stackrel{\text{def}}{=}\frac{a + d}{n} \tag{34}\]
Source: Cohen [7].
Example 10 (Estimated overall agreement for the COVID-19 test) For the agreement table of Example 8, the estimated overall agreement is:
\[ \begin{aligned} \widehat{p}_o &= \frac{693 + 9{,}207}{10{,}000} && \text{(definition of } \widehat{p}_o \text{, with the table's counts)} \\ &= \frac{9{,}900}{10{,}000} && \text{(add)} \\ &= 0.99 && \text{(divide)} \end{aligned} \tag{35}\]
This estimate equals the overall agreement \(p_o = 0.99\) of Example 9, because the table’s cell sample proportions \(a / n = 0.0693\) and \(d / n = 0.9207\) equal the joint probabilities \({\Pr\mathopen{}\left({+}\cap D\right)\mathclose{}}\) and \({\Pr\mathopen{}\left({-}\cap \neg D\right)\mathclose{}}\).
Definition 15 (Positive agreement) The positive agreement (also called the average positive agreement) of an agreement table (Definition 12) with \(2a + b + c > 0\) is the number of positive classifications the agent and the reference share, counted once for each classifier, as a proportion of all their positive classifications:
\[p_{\text{pos}} \stackrel{\text{def}}{=}\frac{2a}{2a + b + c} \tag{36}\]
Source: Cicchetti and Feinstein [8].
Example 11 (Positive agreement for the COVID-19 test) For the agreement table of Example 8, the positive agreement (Definition 15) is:
\[ \begin{aligned} p_{\text{pos}} &= \frac{2 \times 693}{2 \times 693 + 93 + 7} && \text{(definition of } p_{\text{pos}} \text{, with the table's counts)} \\ &= \frac{1{,}386}{1{,}386 + 93 + 7} && \text{(multiply)} \\ &= \frac{1{,}386}{1{,}486} && \text{(add)} \\ &\approx 0.933 && \text{(divide)} \end{aligned} \tag{37}\]
The positive agreement is lower than the overall agreement (Example 9), because the 93 false positives are large relative to the 693 true positives.
Definition 16 (Negative agreement) The negative agreement (also called the average negative agreement) of an agreement table (Definition 12) with \(2d + b + c > 0\) is the number of negative classifications the agent and the reference share, counted once for each classifier, as a proportion of all their negative classifications:
\[p_{\text{neg}} \stackrel{\text{def}}{=}\frac{2d}{2d + b + c} \tag{38}\]
Source: Cicchetti and Feinstein [8].
Example 12 (Negative agreement for the COVID-19 test) For the agreement table of Example 8, the negative agreement (Definition 16) is:
\[ \begin{aligned} p_{\text{neg}} &= \frac{2 \times 9{,}207}{2 \times 9{,}207 + 93 + 7} && \text{(definition of } p_{\text{neg}} \text{, with the table's counts)} \\ &= \frac{18{,}414}{18{,}414 + 93 + 7} && \text{(multiply)} \\ &= \frac{18{,}414}{18{,}514} && \text{(add)} \\ &\approx 0.995 && \text{(divide)} \end{aligned} \tag{39}\]
The negative agreement is even higher than the overall agreement (Example 9), because the 100 disagreements are small relative to the 9,207 shared negative calls.
Definition 17 (Agent-only rate) The agent-only rate of an agreement table (Definition 12) is the proportion of observations that the agent calls positive and the reference calls negative:
\[p_{\text{agent}} \stackrel{\text{def}}{=}\frac{b}{n} \tag{40}\]
Example 13 (Agent-only rate for the COVID-19 test) For the agreement table of Example 8, the agent-only rate (Definition 17) is:
\[ \begin{aligned} p_{\text{agent}} &= \frac{93}{10{,}000} && \text{(definition of } p_{\text{agent}} \text{, with the table's counts)} \\ &= 0.0093 && \text{(divide)} \end{aligned} \tag{41}\]
so about 9 of every 1,000 people tested are called positive by the test and negative by the reference.
Definition 18 (Test measures estimated from an agreement table) Treating the reference as the truth, an agreement table (Definition 12) estimates four measures of the agent:
Each estimate is the matching proportion of the table’s cells, defined when its denominator is positive:
\[ \begin{aligned} \widehat{\text{sensitivity}} &\stackrel{\text{def}}{=}\frac{a}{a + c}, \\ \widehat{\text{specificity}} &\stackrel{\text{def}}{=}\frac{d}{b + d}, \\ \widehat{\text{PPV}} &\stackrel{\text{def}}{=}\frac{a}{a + b}, \\ \widehat{\text{NPV}} &\stackrel{\text{def}}{=}\frac{d}{c + d}. \end{aligned} \tag{42}\]
Example 14 (Test measures for the COVID-19 test) For the agreement table of Example 8, Definition 18 gives:
These estimates match the test’s sensitivity (0.99) and specificity (0.99), its positive predictive value (about 0.88, Example 5), and its negative predictive value (about 0.999, Example 6), because the table’s counts are the expected counts for those values.
Definition 19 (Harmonic mean) The harmonic mean of two positive numbers \(x\) and \(y\) is the reciprocal of the average of their reciprocals:
\[H(x, y) \stackrel{\text{def}}{=}\frac{2}{\frac{1}{x} + \frac{1}{y}} \tag{43}\]
Source: Wikipedia contributors [9].
Example 15 (Harmonic mean of 0.5 and 1) By Definition 19:
\[ \begin{aligned} H(0.5, 1) &= \frac{2}{\frac{1}{0.5} + \frac{1}{1}} && \text{(definition of } H \text{)} \\ &= \frac{2}{2 + 1} && \text{(take each reciprocal)} \\ &= \frac{2}{3} && \text{(add)} \\ &\approx 0.667 && \text{(divide)} \end{aligned} \tag{44}\]
The ordinary average of 0.5 and 1 is 0.75; the harmonic mean sits closer to the smaller number.
Exercise 5 (Positive agreement from PPV and sensitivity) For an agreement table (Definition 12) with \(a > 0\), write the harmonic mean of the estimated positive predictive value and sensitivity (Definition 18) in terms of the table’s cells.
Solution 5. Because \(a > 0\), both \(a + b\) and \(a + c\) are positive, so both estimates are defined and positive:
\[ \begin{aligned} H\mathopen{}\left(\widehat{\text{PPV}}, \widehat{\text{sensitivity}}\right)\mathclose{} &= \frac{2}{\frac{1}{\widehat{\text{PPV}}} + \frac{1}{\widehat{\text{sensitivity}}}} && \text{(definition of } H \text{)} \\ &= \frac{2}{\frac{1}{a / (a + b)} + \frac{1}{\widehat{\text{sensitivity}}}} && \text{(definition of } \widehat{\text{PPV}} \text{)} \\ &= \frac{2}{\frac{1}{a / (a + b)} + \frac{1}{a / (a + c)}} && \text{(definition of } \widehat{\text{sensitivity}} \text{)} \\ &= \frac{2}{\frac{a + b}{a} + \frac{1}{a / (a + c)}} && \text{(reciprocal of a fraction)} \\ &= \frac{2}{\frac{a + b}{a} + \frac{a + c}{a}} && \text{(reciprocal of a fraction)} \\ &= \frac{2}{\frac{(a + b) + (a + c)}{a}} && \text{(add fractions with a common denominator)} \\ &= \frac{2}{\frac{a + b + a + c}{a}} && \text{(remove parentheses)} \\ &= \frac{2}{\frac{a + a + b + c}{a}} && \text{(reorder terms)} \\ &= \frac{2}{\frac{2a + b + c}{a}} && \text{(} a + a = 2a \text{)} \\ &= 2 \cdot\frac{a}{2a + b + c} && \text{(dividing by a fraction multiplies by its reciprocal)} \\ &= \frac{2a}{2a + b + c} && \text{(multiply)} \end{aligned} \tag{45}\]
Theorem 5 (Positive agreement is a harmonic mean) For an agreement table with \(a > 0\), the positive agreement (Definition 15) is the harmonic mean (Definition 19) of the estimated positive predictive value and sensitivity (Definition 18):
\[ p_{\text{pos}} = H\mathopen{}\left(\widehat{\text{PPV}}, \widehat{\text{sensitivity}}\right)\mathclose{} \tag{46}\]
Proof. By Exercise 5, the harmonic mean equals \(2a / (2a + b + c)\), which is \(p_{\text{pos}}\) by Definition 15.
Example 16 (Positive agreement of the COVID-19 test as a harmonic mean) For the agreement table of Example 8, the estimated positive predictive value is \(693 / 786\) and the estimated sensitivity is \(693 / 700\) (Example 14). Theorem 5 gives:
\[ \begin{aligned} p_{\text{pos}} &= \frac{2}{\frac{1}{693 / 786} + \frac{1}{693 / 700}} && \text{(harmonic mean of the two estimates)} \\ &= \frac{2}{\frac{786}{693} + \frac{1}{693 / 700}} && \text{(reciprocal of a fraction)} \\ &= \frac{2}{\frac{786}{693} + \frac{700}{693}} && \text{(reciprocal of a fraction)} \\ &= \frac{2}{\frac{1{,}486}{693}} && \text{(add fractions with a common denominator)} \\ &= 2 \cdot\frac{693}{1{,}486} && \text{(dividing by a fraction multiplies by its reciprocal)} \\ &= \frac{1{,}386}{1{,}486} && \text{(multiply)} \\ &\approx 0.933 && \text{(divide)} \end{aligned} \tag{47}\]
which matches Example 11. The lower positive predictive value (about 0.88) pulls the positive agreement down from the sensitivity (0.99), as in Example 15.
Exercise 6 (Negative agreement from NPV and specificity) For an agreement table (Definition 12) with \(d > 0\), write the harmonic mean of the estimated negative predictive value and specificity (Definition 18) in terms of the table’s cells.
Solution 6. Because \(d > 0\), both \(c + d\) and \(b + d\) are positive, so both estimates are defined and positive:
\[ \begin{aligned} H\mathopen{}\left(\widehat{\text{NPV}}, \widehat{\text{specificity}}\right)\mathclose{} &= \frac{2}{\frac{1}{\widehat{\text{NPV}}} + \frac{1}{\widehat{\text{specificity}}}} && \text{(definition of } H \text{)} \\ &= \frac{2}{\frac{1}{d / (c + d)} + \frac{1}{\widehat{\text{specificity}}}} && \text{(definition of } \widehat{\text{NPV}} \text{)} \\ &= \frac{2}{\frac{1}{d / (c + d)} + \frac{1}{d / (b + d)}} && \text{(definition of } \widehat{\text{specificity}} \text{)} \\ &= \frac{2}{\frac{c + d}{d} + \frac{1}{d / (b + d)}} && \text{(reciprocal of a fraction)} \\ &= \frac{2}{\frac{c + d}{d} + \frac{b + d}{d}} && \text{(reciprocal of a fraction)} \\ &= \frac{2}{\frac{(c + d) + (b + d)}{d}} && \text{(add fractions with a common denominator)} \\ &= \frac{2}{\frac{c + d + b + d}{d}} && \text{(remove parentheses)} \\ &= \frac{2}{\frac{d + d + b + c}{d}} && \text{(reorder terms)} \\ &= \frac{2}{\frac{2d + b + c}{d}} && \text{(} d + d = 2d \text{)} \\ &= 2 \cdot\frac{d}{2d + b + c} && \text{(dividing by a fraction multiplies by its reciprocal)} \\ &= \frac{2d}{2d + b + c} && \text{(multiply)} \end{aligned} \tag{48}\]
Theorem 6 (Negative agreement is a harmonic mean) For an agreement table with \(d > 0\), the negative agreement (Definition 16) is the harmonic mean (Definition 19) of the estimated negative predictive value and specificity (Definition 18):
\[ p_{\text{neg}} = H\mathopen{}\left(\widehat{\text{NPV}}, \widehat{\text{specificity}}\right)\mathclose{} \tag{49}\]
Proof. By Exercise 6, the harmonic mean equals \(2d / (2d + b + c)\), which is \(p_{\text{neg}}\) by Definition 16.
Example 17 (Negative agreement of the COVID-19 test as a harmonic mean) For the agreement table of Example 8, the estimated negative predictive value is \(9{,}207 / 9{,}214\) and the estimated specificity is \(9{,}207 / 9{,}300\) (Example 14). Theorem 6 gives:
\[ \begin{aligned} p_{\text{neg}} &= \frac{2}{\frac{1}{9{,}207 / 9{,}214} + \frac{1}{9{,}207 / 9{,}300}} && \text{(harmonic mean of the two estimates)} \\ &= \frac{2}{\frac{9{,}214}{9{,}207} + \frac{1}{9{,}207 / 9{,}300}} && \text{(reciprocal of a fraction)} \\ &= \frac{2}{\frac{9{,}214}{9{,}207} + \frac{9{,}300}{9{,}207}} && \text{(reciprocal of a fraction)} \\ &= \frac{2}{\frac{18{,}514}{9{,}207}} && \text{(add fractions with a common denominator)} \\ &= 2 \cdot\frac{9{,}207}{18{,}514} && \text{(dividing by a fraction multiplies by its reciprocal)} \\ &= \frac{18{,}414}{18{,}514} && \text{(multiply)} \\ &\approx 0.995 && \text{(divide)} \end{aligned} \tag{50}\]
which matches Example 12.
Definition 20 (Chance agreement) Let:
The chance agreement of the agent and the reference on that population is the overall agreement they would have if \(A\) and \(R\) were statistically independent with their marginal probabilities \({\Pr\mathopen{}\left(A\right)\mathclose{}}\) and \({\Pr\mathopen{}\left(R\right)\mathclose{}}\):
\[p_e \stackrel{\text{def}}{=}{\Pr\mathopen{}\left(A\right)\mathclose{}} \cdot{\Pr\mathopen{}\left(R\right)\mathclose{}} + {\Pr\mathopen{}\left(\neg A\right)\mathclose{}} \cdot{\Pr\mathopen{}\left(\neg R\right)\mathclose{}} \tag{51}\]
Source: Cohen [7].
Example 18 (Chance agreement for the COVID-19 test) Apply the COVID-19 test of Example 2 and Example 3 to the population of Example 4, with the test as the agent (\(A = {+}\)) and the disease status as the reference (\(R = D\)). By Theorem 2, with sensitivity \(\text{sens} = 0.99\), specificity \(\text{spec} = 0.99\), and prevalence \(\pi= {\Pr\mathopen{}\left(D\right)\mathclose{}} = 0.07\), the probability of a positive test is:
\[ \begin{aligned} {\Pr\mathopen{}\left({+}\right)\mathclose{}} &= \text{sens} \cdot\pi+ \mathopen{}\left(1 - \text{spec}\right)\mathclose{} \cdot\mathopen{}\left(1 - \pi\right)\mathclose{} && \text{(probability of a positive test)} \\ &= 0.99 \cdot 0.07 + \mathopen{}\left(1 - 0.99\right)\mathclose{} \cdot\mathopen{}\left(1 - 0.07\right)\mathclose{} && \text{(substitute the given values)} \\ &= 0.99 \cdot 0.07 + 0.01 \cdot 0.93 && \text{(subtract inside each pair of parentheses)} \\ &= 0.0693 + 0.0093 && \text{(multiply each product)} \\ &= 0.0786 && \text{(add)} \end{aligned} \tag{52}\]
By the complement rule and Theorem 1, the probability of a negative test is:
\[ \begin{aligned} {\Pr\mathopen{}\left({-}\right)\mathclose{}} &= {\Pr\mathopen{}\left(\neg {+}\right)\mathclose{}} && \text{(negative classification is the complement of positive)} \\ &= 1 - {\Pr\mathopen{}\left({+}\right)\mathclose{}} && \text{(complement rule)} \\ &= 1 - 0.0786 && \text{(substitute } {\Pr\mathopen{}\left({+}\right)\mathclose{}} \text{)} \\ &= 0.9214 && \text{(subtract)} \end{aligned} \tag{53}\]
With \({\Pr\mathopen{}\left(D\right)\mathclose{}} = 0.07\) and \({\Pr\mathopen{}\left(\neg D\right)\mathclose{}} = 0.93\) (Example 4), the chance agreement is:
\[ \begin{aligned} p_e &= {\Pr\mathopen{}\left({+}\right)\mathclose{}} \cdot{\Pr\mathopen{}\left(D\right)\mathclose{}} + {\Pr\mathopen{}\left({-}\right)\mathclose{}} \cdot{\Pr\mathopen{}\left(\neg D\right)\mathclose{}} && \text{(definition of } p_e \text{)} \\ &= 0.0786 \cdot 0.07 + 0.9214 \cdot 0.93 && \text{(substitute the marginal probabilities)} \\ &= 0.005502 + 0.856902 && \text{(multiply each product)} \\ &= 0.862404 && \text{(add)} \end{aligned} \tag{54}\]
Because nearly everyone tests negative and does not have COVID-19, two independent classifiers with \({\Pr\mathopen{}\left({+}\right)\mathclose{}} = 0.0786\) and \({\Pr\mathopen{}\left(D\right)\mathclose{}} = 0.07\) would agree on about 86% of people by chance alone.
Definition 21 (Estimated chance agreement) The estimated chance agreement of an agreement table estimates the chance agreement \(p_e\) by replacing the marginal probabilities \({\Pr\mathopen{}\left(A\right)\mathclose{}}\), \({\Pr\mathopen{}\left(R\right)\mathclose{}}\), \({\Pr\mathopen{}\left(\neg A\right)\mathclose{}}\), and \({\Pr\mathopen{}\left(\neg R\right)\mathclose{}}\) with the table’s marginal sample proportions \((a + b) / n\), \((a + c) / n\), \((c + d) / n\), and \((b + d) / n\):
\[\widehat{p}_e \stackrel{\text{def}}{=}\frac{(a + b)(a + c) + (c + d)(b + d)}{n^2} \tag{55}\]
Source: Cohen [7].
Example 19 (Estimated chance agreement for the COVID-19 test) For the agreement table of Example 8, the estimated chance agreement is:
\[ \begin{aligned} \widehat{p}_e &= \frac{786 \times 700 + 9{,}214 \times 9{,}300}{10{,}000^2} && \text{(definition of } \widehat{p}_e \text{, with the table's counts)} \\ &= \frac{550{,}200 + 85{,}690{,}200}{10{,}000^2} && \text{(multiply)} \\ &= \frac{86{,}240{,}400}{10{,}000^2} && \text{(add)} \\ &= \frac{86{,}240{,}400}{100{,}000{,}000} && \text{(square)} \\ &= 0.862404 && \text{(divide)} \end{aligned} \tag{56}\]
This estimate equals the chance agreement \(p_e = 0.862404\) of Example 18, because the table’s marginal sample proportions equal the marginal probabilities \({\Pr\mathopen{}\left({+}\right)\mathclose{}}\), \({\Pr\mathopen{}\left(D\right)\mathclose{}}\), \({\Pr\mathopen{}\left({-}\right)\mathclose{}}\), and \({\Pr\mathopen{}\left(\neg D\right)\mathclose{}}\).
Definition 22 (Cohen’s kappa) For an agent and a reference with chance agreement \(p_e < 1\), Cohen’s kappa is the agreement beyond chance, as a fraction of the largest possible agreement beyond chance, built from the overall agreement \(p_o\) and the chance agreement \(p_e\):
\[\kappa \stackrel{\text{def}}{=}\frac{p_o - p_e}{1 - p_e} \tag{57}\]
Example 20 (Cohen’s kappa for the COVID-19 test) For the COVID-19 test of Example 2 and Example 3 in the population of Example 4, the overall agreement is \(p_o = 0.99\) (Example 9) and the chance agreement is \(p_e = 0.862404\) (Example 18), so Cohen’s kappa is:
\[ \begin{aligned} \kappa &= \frac{0.99 - 0.862404}{1 - 0.862404} && \text{(definition of } \kappa \text{)} \\ &= \frac{0.127596}{1 - 0.862404} && \text{(subtract in the numerator)} \\ &= \frac{0.127596}{0.137596} && \text{(subtract in the denominator)} \\ &\approx 0.927 && \text{(divide)} \end{aligned} \tag{58}\]
Definition 23 (Estimated Cohen’s kappa) For an agreement table with estimated chance agreement \(\widehat{p}_e < 1\), the estimated Cohen’s kappa estimates Cohen’s kappa \(\kappa\) from the estimated overall agreement \(\widehat{p}_o\) and the estimated chance agreement \(\widehat{p}_e\):
\[\hat{\kappa}\stackrel{\text{def}}{=}\frac{\widehat{p}_o - \widehat{p}_e}{1 - \widehat{p}_e} \tag{59}\]
Example 21 (Estimated Cohen’s kappa for the COVID-19 test) For the agreement table of Example 8, the estimated overall agreement is \(\widehat{p}_o = 0.99\) (Example 10) and the estimated chance agreement is \(\widehat{p}_e = 0.862404\) (Example 19), so the estimated Cohen’s kappa is:
\[ \begin{aligned} \hat{\kappa} &= \frac{0.99 - 0.862404}{1 - 0.862404} && \text{(definition of } \hat{\kappa}\text{)} \\ &= \frac{0.127596}{1 - 0.862404} && \text{(subtract in the numerator)} \\ &= \frac{0.127596}{0.137596} && \text{(subtract in the denominator)} \\ &\approx 0.927 && \text{(divide)} \end{aligned} \tag{60}\]
This estimate equals Cohen’s kappa \(\kappa \approx 0.927\) of Example 20, because \(\widehat{p}_o = p_o\) and \(\widehat{p}_e = p_e\).
Example 22 (High agreement but low kappa) Now apply the same test to \(n = 100{,}000\) people in a population where the prevalence is only 0.5%. Then:
agree_rare <- c(a = 495, b = 995, c = 5, d = 98505)
rare_metrics <- with(as.list(agree_rare), c(
overall = (a + d) / (a + b + c + d),
positive = 2 * a / (2 * a + b + c),
negative = 2 * d / (2 * d + b + c),
kappa = kappa_from_table(a, b, c, d)
))
round(rare_metrics, 4)
#> overall positive negative kappa
#> 0.9900 0.4975 0.9949 0.4937The estimated overall agreement is still 0.99, but the estimated Cohen’s kappa drops to about 0.49 and the positive agreement to about 0.50, while the negative agreement stays at about 0.995. The positive and negative agreements show where the classifiers disagree: the test and the reference agree on negatives and disagree on about half of the positive calls.
Source: Feinstein and Cicchetti [11] (the first paradox).
Remark 3 (Kappa depends on prevalence). When both classifiers call one class rarely, the chance agreement \(p_e\) is close to 1, so the same overall agreement leaves little room for agreement beyond chance, and Cohen’s kappa can be low even when the overall agreement is high, as Example 22 shows. Cicchetti and Feinstein [8] recommend reporting the positive agreement and negative agreement alongside Cohen’s kappa, so that poor agreement on the rarer class is not hidden.
Source: Feinstein and Cicchetti [11]; Cicchetti and Feinstein [8].