Predicting a category
Linear regression predicts a number. A great many questions are not about a number: whether a patient recovers, whether an email is spam, which of ten digits a scan shows.
Let the data be \(\mathcal{X} = \set{(\vx_1, y_1), \ldots, (\vx_n, y_n)}\) with \(\vx_i \in \reals^{\ninputvars}\) and \(y_i\) a label rather than a measurement. Two labels is binary classification; more than two is multi-class classification.
We could predict the label directly, with a classification rule that maps each feature vector to one of the classes. Predicting \(\Prf{Y = 1 \mid \vx}\) gives more information. A predicted class label names only a class; a predicted probability also measures how likely each class is, so whoever uses the prediction can choose how to act on it, for instance with a threshold that reflects the cost of each kind of error.
Decision regions and boundaries
A classification rule splits the set of all feature vectors into pieces, one for each class that it predicts.
Example 1 (Decision regions of a blood-pressure rule) A cohort study, the Western Collaborative Group Study [2], followed 3154 middle-aged men for coronary heart disease, and recorded each man’s systolic blood pressure \(\predvar\), in mmHg. Let the classification rule \(c\) predict heart disease (\(c(\predvar) = 1\)) when \(\predvar \ge 140\), and no heart disease (\(c(\predvar) = 0\)) otherwise. We chose the cutoff 140 mmHg by hand, as a round number in the upper part of the data; logistic regression instead fits a classifier to data. The decision region of class \(1\) is \(\setbuild{\predvar}{\predvar \ge 140}\), and the decision region of class \(0\) is \(\setbuild{\predvar}{\predvar < 140}\). 693 of the men are in the first region, and 2461 are in the second. Figure 1 shades the two regions over the men’s blood pressures, with one bar for each recorded pressure.
Example 2 (The men who had heart disease are not a decision region) In the Western Collaborative Group Study data of Example 1, 257 men had coronary heart disease. The set of their blood pressures is not the decision region of class \(1\) under the rule \(c\) of Example 1, because a decision region depends on what \(c\) predicts, not on the observed labels. Only 96 of those men are in the decision region of class \(1\); the other 161 had blood pressures below 140 mmHg, so \(c\) predicts no heart disease for them.
Example 3 (Decision boundary of a blood-pressure rule) The rule \(c\) of Example 1 predicts heart disease when the systolic blood pressure \(\predvar\) is at least 140 mmHg. Its decision boundary is the single point \(\predvar = 140\), which lies between the decision region \(\setbuild{\predvar}{\predvar < 140}\) and the decision region \(\setbuild{\predvar}{\predvar \ge 140}\). It is the dashed line in Figure 1.
Example 4 (A cutoff that is not a decision boundary) The point \(\predvar = 120\) is not on the decision boundary of the rule \(c\) of Example 1. Blood pressures slightly below and slightly above 120 mmHg are both in the decision region of class \(0\), so the point does not separate two decision regions.
How often a rule is wrong
The error rate of a classification rule counts how often it misclassifies. Example 5 and Example 6 compute it for two rules on the men of the Western Collaborative Group Study.
Example 5 (Error rate of a blood-pressure rule) The rule \(c\) of Example 1 makes two kinds of mistake on the 3154 men. It predicts heart disease for 597 men who had none, and predicts no heart disease for 161 men who had it. Its error rate on these men is therefore
\[
\frac{758}{3154} \approx 0.240.
\]
Example 6 (A rule that never predicts heart disease) A second rule \(c_0\) predicts no heart disease for every man, so \(c_0(\predvar) = 0\) whatever \(\predvar\) is. It is wrong only for the 257 men who had heart disease, so its error rate on the 3154 men is
\[
\frac{257}{3154} \approx 0.081.
\]
This error rate is lower than that of the rule \(c\) in Example 5, even though \(c_0\) never finds a single case of heart disease. Heart disease is rare in these data, so the error rate alone favors a rule that ignores blood pressure. The Metrics page reports each kind of mistake separately.
References
[2]
R. H. Rosenman, R. J. Brand, C. D. Jenkins, M. Friedman, R. Straus, and M. Wurm,
“Coronary heart disease in the Western Collaborative Group Study: Final follow-up experience of 8 1/2 years,” JAMA, vol. 233, no. 8, pp. 872–877, 1975, doi:
10.1001/jama.1975.03260080034016.
Back to top