Last modified: 2026-11-07 21:34:29 (PST)
Definition 1 (Scientific models) Scientific models are attempts to describe physical conditions or changes that occur in the world and universe around us.
Example 1 (Scientific models in epidemiology) Epidemiologists typically study biological conditions and changes, such as the spread of infectious diseases through populations, or the effects of environmental factors on individuals.
…Essentially, all models are wrong, but some are useful. However, the approximate nature of the model must always be borne in mind.
[Box and Draper [1], p. 424; emphasis added]
When we perform statistical analyses, we use data to help us choose between models; specifically, to determine which models best explain those data.
Physical processes do not produce data on their own. Data are only produced when scientists implement an observation process (that is, a scientific study), which is distinct from the underlying physical process. In some cases, the observation process and the physical process interact with each other; this interaction is called the “observer effect”.
To learn about the physical processes we are ultimately interested in, we often need to account for the observation process that produced the data we are analyzing. In particular, if some of the planned observations in the study design were not completed, we will likely need to account for the incompleteness of the resulting data set in our analysis. If we are not sure why some observations are incomplete, we may need to model the observation process in addition to the physical process we were originally interested in. For example, if some participants in a study dropped out part-way through, we may need to investigate why those participants dropped out, as opposed to other participants who completed the study.
These kinds of missing data issues are outside the scope of these notes; see Van Buuren [3] for more details.
Example 2 (Mean height of students) If we are trying to determine the population mean height of students at our school, then the population mean is our estimand.
In statistical contexts, most estimands are parameters of probabilistic models, or functions of model parameters.
Notation for estimands
Model parameters and other estimands are often symbolized using lower-case Greek letters. The notation choices page gives our convention and the alternatives.
Definition 3 (Estimate/estimated value) In statistics, an estimate or estimated value \(\hat{\theta}\) is an informed guess of an estimand \(\theta\)’s value, computed from the observed data \(x_1, \ldots, x_n\):
\[ \hat{\theta}= \hat{\theta}(x_1, \ldots, x_n). \]
Example 3 (Mean height of students) Suppose we measure the heights of 50 randomly sampled students from our school, and their sample mean is 175 cm. We might use 175 cm as an estimate of the population mean.
Definition 4 (Estimator) An estimator is the function \(\hat{\theta}(\cdot)\) that transforms data into an estimate; applied to a random sample \(X_1, \ldots, X_n\), it is a random variable:
\[ \hat{\theta}(X_1, \ldots, X_n). \]
Source: Wikipedia contributors [7].
Estimators are random variables
When an estimator is applied to random variables \(X_1, \ldots, X_n\) rather than to their observed values, the result \(\hat{\theta}(X_1, \ldots, X_n)\) is also a random variable.
Notation for estimators
Estimators are often symbolized by placing a hat over the symbol of the corresponding estimand; for example, \(\hat{\theta}\). The notation choices page gives our convention and the alternatives.
Usually, their dependence on the data is implicit:
\[\hat{\theta}\stackrel{\text{def}}{=}\hat{\theta}(x_1, \ldots, x_n)\]
Example 4 (Mean height of students) Suppose we want to estimate the population mean height of students at our school, which we will represent as \(\mu\), and we measure the heights of \(n = 50\) randomly sampled students as random variables \(X_1, \ldots, X_n\). Then we could use the function
\[\hat{\mu}(X_1, \ldots, X_n) \stackrel{\text{def}}{=}\frac{1}{n}\sum_{i=1}^nX_i\stackrel{\text{def}}{=}\bar X\]
as an estimator to produce an estimate \(\hat{\mu}= \bar x\) of \(\mu\).
Another estimator would be just the height of the first student sampled:
\[\hat{\mu}^{(2)}(X_1, \ldots, X_n) \stackrel{\text{def}}{=}X_1\]
A third possible estimator would be the sample mean of all sampled students’ heights, except for the two most extreme. Using the order statistics \(X_{(1)} \le X_{(2)} \le \cdots \le X_{(n)}\) (the observations sorted in increasing order), this estimator drops \(X_{(1)}\) and \(X_{(n)}\) and averages the remaining \(n - 2\) observations:
\[\hat{\mu}^{(3)}(X_1, \ldots, X_n) \stackrel{\text{def}}{=}\frac{1}{n-2}\sum_{i=2}^{n-1} X_{(i)}\]
Which of these estimators is best? The answer depends on how we evaluate them (see Section 3).
Definition 5 (Sampling distribution) The sampling distribution of an estimator \(\hat{\theta}\) is the probability distribution of \(\hat{\theta}(X_1, \ldots, X_n)\) over repeated random samples \(X_1, \ldots, X_n\) from the same population: the distribution of the values the estimator would take across all the samples we might have drawn.
Source: Wikipedia contributors [8].
Example 5 (Sampling distribution of a sample mean) Suppose student heights are \(X_i\ \sim_{\operatorname{iid}}\ {\operatorname{N}\mathopen{}\left(\mu, \sigma^2\right)\mathclose{}}\) with standard deviation \(\sigma= 10\) cm, and we sample \(n = 25\) students. The sample mean \(\bar X\) is a linear combination of independent Gaussian variables, so it is also Gaussian, with mean \({\mathbb{E}\mathopen{}\left[\bar X\right]\mathclose{}} = \mu\) and variance \({\operatorname{Var}\mathopen{}\left(\bar X\right)\mathclose{}} = \sigma^2/ n\). Its sampling distribution is therefore
\[\bar X \sim {\operatorname{N}\mathopen{}\left(\mu, \sigma^2/ 25\right)\mathclose{}},\]
which has standard deviation \(\sigma/ \sqrt{25} = 10 / 5 = 2\) cm. Across repeated samples of 25 students, about 95% of the sample means land within \(2 \times 2 = 4\) cm of the true mean \(\mu\).
It’s helpful to keep in mind the mathematical type of each estimation concept:
To determine which estimator is best, we need to define best. Accuracy is usually most important, and ease of computation is usually secondary.
Definition 6 (Estimation error) The estimation error of an estimate \(\hat{\theta}\) of a true value \(\theta\) is the difference between the estimate and the estimand \(\theta\):
\[{\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}} \stackrel{\text{def}}{=}\hat{\theta}- \theta\]
Example 6 (Estimation error of a mean height) Continuing Example 3, suppose the population mean height of students at our school were actually \(\mu= 172\) cm. Then the estimate \(\hat{\mu}= 175\) cm would have estimation error:
\[ \begin{aligned} {\varepsilon\mathopen{}\left(\hat{\mu}\right)\mathclose{}} &= \hat{\mu}- \mu && \text{(definition of estimation error)}\\ &= 175 - 172 && \text{(substitute the estimate and the estimand)}\\ &= 3 \text{ cm} && \text{(subtract)} \end{aligned} \]
In practice we never observe an estimation error, because we do not know the estimand’s value; if we did, we would not need to estimate it.
The accuracy of an estimator has no single, agreed formal definition. The usual measures of accuracy, including the bias, population mean squared error, and population mean absolute error defined in this section, are all functions of the distribution of the estimator’s estimation error.
Predictions, fitted values, residuals, prediction errors and the sample mean squared error of predictions have their own page, Prediction. That page also contrasts them with the estimation quantities defined here: estimation error and the population mean squared error of an estimator.
Definition 7 (Bias) The bias of an estimator \(\hat{\theta}\) for an estimand \(\theta\) is the expected value of the estimation error:
\[{\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[{\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right]\mathclose{}} \tag{1}\]
Source: Wikipedia contributors [12].
Exercise 1 (Bias in terms of the expectation) Derive an expression for the bias \({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\) of an estimator \(\hat{\theta}\) in terms of its expected value \({\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\) and the estimand \(\theta\).
Solution 1. \[ \begin{aligned} {\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} &\stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[{\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right]\mathclose{}} && \text{(definition of bias)}\\ &= {\mathbb{E}\mathopen{}\left[\hat{\theta}- \theta\right]\mathclose{}} && \text{(definition of estimation error)}\\ &= {\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}} - {\mathbb{E}\mathopen{}\left[\theta\right]\mathclose{}} && \text{(linearity of expectation)}\\ &= {\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}} - \theta && \text{($\theta$ is a constant)} \end{aligned} \]
Theorem 1 (Bias equals expectation minus truth) \[{\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}} - \theta\]
Proof. This is the solution to Exercise 1.
Example 7 (Bias of two estimators of a population mean) Let \(X_1, \ldots, X_n\) each have expectation \({\mathbb{E}\mathopen{}\left[X_i\right]\mathclose{}} = \mu\), as in Example 4. By Theorem 1, the bias of the sample mean \(\bar X\) is:
\[ \begin{aligned} {\operatorname{Bias}\mathopen{}\left(\bar X\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[\bar X\right]\mathclose{}} - \mu && \text{(bias equals expectation minus truth)}\\ &= {\mathbb{E}\mathopen{}\left[\frac{1}{n}\sum_{i=1}^nX_i\right]\mathclose{}} - \mu && \text{(definition of $\bar X$)}\\ &= \frac{1}{n}{\mathbb{E}\mathopen{}\left[\sum_{i=1}^nX_i\right]\mathclose{}} - \mu && \text{(factor the constant $\tfrac{1}{n}$ out of the expectation)}\\ &= \frac{1}{n}\sum_{i=1}^n{\mathbb{E}\mathopen{}\left[X_i\right]\mathclose{}} - \mu && \text{(linearity of expectation)}\\ &= \frac{1}{n}\sum_{i=1}^n\mu- \mu && \text{(${\mathbb{E}\mathopen{}\left[X_i\right]\mathclose{}} = \mu$ for every $i$)}\\ &= \frac{1}{n} \cdot n\mu- \mu && \text{(sum of $n$ copies of $\mu$)}\\ &= \mu- \mu && \text{(cancel $n$)}\\ &= 0 && \text{(subtract)} \end{aligned} \]
and the bias of the single-observation estimator \(\hat{\mu}^{(2)} = X_1\) is:
\[ \begin{aligned} {\operatorname{Bias}\mathopen{}\left(X_1\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[X_1\right]\mathclose{}} - \mu && \text{(bias equals expectation minus truth)}\\ &= \mu- \mu && \text{(${\mathbb{E}\mathopen{}\left[X_1\right]\mathclose{}} = \mu$)}\\ &= 0 && \text{(subtract)} \end{aligned} \]
So both estimators have zero bias, even though \(\bar X\) uses all \(n\) observations and \(X_1\) uses only one.
Definition 8 (Mean squared error) The population mean squared error (pMSE) of an estimator \(\hat{\theta}\), also called its mean squared error or its expected squared estimation error (ESEE), denoted \({\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\) or \({\operatorname{ESEE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\), is the expectation of the square of the estimation error:
\[{\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[{\mathopen{}\left({\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}}\]
Source: Wikipedia contributors [13].
Exercise 2 (Expanding the squared bias) Using Theorem 1, expand the squared bias \({\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2}\) in terms of \({\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\) and the estimand \(\theta\).
Solution 2. \[ \begin{aligned} {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} &= {\mathopen{}\left({\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}} - \theta\right)\mathclose{}}^{2} && \text{(bias equals expectation minus truth)}\\ &= {\mathopen{}\left({\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\right)\mathclose{}}^{2} - 2{\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\theta+ \theta^2 && \text{(expand the binomial square)} \end{aligned} \]
Theorem 2 (Expanded squared bias) For any one-dimensional estimator \(\hat{\theta}\) of an estimand \(\theta\), the squared bias is:
\[{\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} = {\mathopen{}\left({\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\right)\mathclose{}}^{2} - 2{\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\theta+ \theta^2\]
Proof. This is the solution to Exercise 2, worked in Solution 2.
Exercise 3 (Squared bias plus variance) Using Exercise 2, write \({\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} + {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\) in terms of \({\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}}\), \({\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\), and the estimand \(\theta\).
Solution 3. The variance is (simplified expression for variance):
\[{\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - {\mathopen{}\left({\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\right)\mathclose{}}^{2} \tag{2}\]
Let the population mean of the estimator \(\hat{\theta}\) be \(\mu_{\hat{\theta}} = {\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\). Then the variance of \(\hat{\theta}\) is \({\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - {\mathopen{}\left(\mu_{\hat{\theta}}\right)\mathclose{}}^{2}\), and the squared bias of \(\hat{\theta}\) for the estimand \(\theta\) from Exercise 2 is \({\mathopen{}\left(\mu_{\hat{\theta}}\right)\mathclose{}}^{2} - 2\mu_{\hat{\theta}}\theta+ \theta^2\). Add them and simplify:
\[ \begin{aligned} {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} + {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} &= {\mathopen{}\left(\mu_{\hat{\theta}}\right)\mathclose{}}^{2} - 2\mu_{\hat{\theta}}\theta+ \theta^2 + {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} && \text{(substitute the squared bias)}\\ &= {\mathopen{}\left(\mu_{\hat{\theta}}\right)\mathclose{}}^{2} - 2\mu_{\hat{\theta}}\theta+ \theta^2 + {\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - {\mathopen{}\left(\mu_{\hat{\theta}}\right)\mathclose{}}^{2} && \text{(substitute the variance)}\\ &= {\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - 2\mu_{\hat{\theta}}\theta+ \theta^2 + {\mathopen{}\left(\mu_{\hat{\theta}}\right)\mathclose{}}^{2} - {\mathopen{}\left(\mu_{\hat{\theta}}\right)\mathclose{}}^{2} && \text{(reorder the terms)}\\ &= {\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - 2\mu_{\hat{\theta}}\theta+ \theta^2 + \mathopen{}\left({\mathopen{}\left(\mu_{\hat{\theta}}\right)\mathclose{}}^{2} - {\mathopen{}\left(\mu_{\hat{\theta}}\right)\mathclose{}}^{2}\right)\mathclose{} && \text{(group the last two terms)}\\ &= {\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - 2\mu_{\hat{\theta}}\theta+ \theta^2 + 0 && \text{($a - a = 0$)}\\ &= {\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - 2\mu_{\hat{\theta}}\theta+ \theta^2 && \text{($a + 0 = a$)} \end{aligned} \tag{3}\]
Theorem 3 (Expanded squared bias plus variance) For any one-dimensional estimator \(\hat{\theta}\) of an estimand \(\theta\), the squared bias plus the variance is:
\[{\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} + {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - 2{\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\theta+ \theta^2\]
Proof. This is the solution to Exercise 3, worked in Solution 3. It adds the variance to the squared bias of Theorem 2.
Exercise 4 (Expanding the population mean squared error) Derive an expression for the pMSE \({\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\) in terms of \({\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}}\), \({\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\), and the estimand \(\theta\).
Solution 4. \[ \begin{aligned} {\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} &\stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[{\mathopen{}\left({\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} && \text{(definition of MSE)}\\ &= {\mathbb{E}\mathopen{}\left[(\hat{\theta}- \theta)^2\right]\mathclose{}} && \text{(definition of estimation error)}\\ &= {\mathbb{E}\mathopen{}\left[\hat{\theta}^2 - 2\hat{\theta}\theta+ \theta^2\right]\mathclose{}} && \text{(expand the binomial square)}\\ &= {\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - {\mathbb{E}\mathopen{}\left[2\hat{\theta}\theta\right]\mathclose{}} + {\mathbb{E}\mathopen{}\left[\theta^2\right]\mathclose{}} && \text{(linearity of expectation)}\\ &= {\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - 2{\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\theta+ {\mathbb{E}\mathopen{}\left[\theta^2\right]\mathclose{}} && \text{(factor the constant $2\theta$ out of the middle expectation)}\\ &= {\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - 2{\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\theta+ \theta^2 && \text{(the expectation of a constant is that constant)} \end{aligned} \]
Theorem 4 (Expanded population mean squared error) For any one-dimensional estimator \(\hat{\theta}\) of an estimand \(\theta\), the pMSE is:
\[{\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = {\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - 2{\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\theta+ \theta^2\]
Proof. This is the solution to Exercise 4, worked in Solution 4.
Theorem 5 (Population mean squared error equals bias squared plus variance) For any one-dimensional estimator \(\hat{\theta}\):
\[{\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} + {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} \tag{4}\]
Proof (Shorter proof). Apply the second moment is variance plus squared mean lemma to the estimation error \(Z = {\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\):
\[ \begin{aligned} {\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[{\mathopen{}\left({\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2}\right]\mathclose{}} && \text{(definition of pMSE)}\\ &= {\operatorname{Var}\mathopen{}\left({\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}} + {\mathopen{}\left({\mathbb{E}\mathopen{}\left[{\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right]\mathclose{}}\right)\mathclose{}}^{2} && \text{(second moment is variance plus squared mean)}\\ &= {\operatorname{Var}\mathopen{}\left({\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}} + {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} && \text{(definition of bias)}\\ &= {\operatorname{Var}\mathopen{}\left(\hat{\theta}- \theta\right)\mathclose{}} + {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} && \text{(definition of estimation error)}\\ &= {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} + {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} && \text{(subtracting a constant does not change a variance)} \end{aligned} \]
Unlike the expansion route, this proof never expands the square.
Source: Wikipedia contributors [13].
Proof. By Theorem 4 and Theorem 3, \({\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\) and \({\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} + {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\) both equal \({\mathbb{E}\mathopen{}\left[\hat{\theta}^2\right]\mathclose{}} - 2{\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}}\theta+ \theta^2\), so they are equal to each other.
Remark 1 (Estimation versus prediction). The squared-error split has two forms, and they differ in what the estimate is compared with.
Example 8 (Population mean squared error of two estimators of a population mean) Continuing Example 7, suppose also that \(X_1, \ldots, X_n\) are mutually independent, each with variance \({\operatorname{Var}\mathopen{}\left(X_i\right)\mathclose{}} = \sigma^2\). Both \(\bar X\) and \(X_1\) have zero bias, so by Theorem 5 each estimator’s pMSE equals its variance.
For \(X_1\),
\[ \begin{aligned} {\operatorname{pMSE}\mathopen{}\left(X_1\right)\mathclose{}} &= 0^2 + {\operatorname{Var}\mathopen{}\left(X_1\right)\mathclose{}} \\ &= \sigma^2. \end{aligned} \]
For \(\bar X\):
\[ \begin{aligned} {\operatorname{pMSE}\mathopen{}\left(\bar X\right)\mathclose{}} &= {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\bar X\right)\mathclose{}}\right)\mathclose{}}^{2} + {\operatorname{Var}\mathopen{}\left(\bar X\right)\mathclose{}} && \text{(pMSE equals bias squared plus variance)}\\ &= 0^2 + {\operatorname{Var}\mathopen{}\left(\bar X\right)\mathclose{}} && \text{(zero bias)}\\ &= {\operatorname{Var}\mathopen{}\left(\bar X\right)\mathclose{}} && \text{($0^2 = 0$)}\\ &= {\operatorname{Var}\mathopen{}\left(\frac{1}{n}\sum_{i=1}^nX_i\right)\mathclose{}} && \text{(definition of $\bar X$)}\\ &= \frac{1}{n^2}\sum_{i=1}^n{\operatorname{Var}\mathopen{}\left(X_i\right)\mathclose{}} && \text{(variance of a linear combination of independent variables)}\\ &= \frac{1}{n^2}\sum_{i=1}^n\sigma^2 && \text{(${\operatorname{Var}\mathopen{}\left(X_i\right)\mathclose{}} = \sigma^2$ for every $i$)}\\ &= \frac{1}{n^2} \cdot n\sigma^2 && \text{(sum of $n$ copies of $\sigma^2$)}\\ &= \frac{\sigma^2}{n} && \text{(cancel one factor of $n$)} \end{aligned} \]
The step to \(\frac{1}{n^2}\sum_{i=1}^n{\operatorname{Var}\mathopen{}\left(X_i\right)\mathclose{}}\) uses the variance of a linear combination, whose covariance terms are all zero for independent variables. The result \(\sigma^2/n\) is the variance of an IID sample mean.
So for any sample size \(n > 1\),
\[ \begin{aligned} {\operatorname{pMSE}\mathopen{}\left(\bar X\right)\mathclose{}} &= \sigma^2/n \\ &< \sigma^2\\ &= {\operatorname{pMSE}\mathopen{}\left(X_1\right)\mathclose{}}: \end{aligned} \]
by mean squared error, the sample mean is the more accurate estimator. With \(n = 50\) students, as in Example 4, the sample mean’s mean squared error is \(1/50\) of \(X_1\)’s.
Definition 9 (Root mean squared error) The population root mean squared error (pRMSE) of an estimator \(\hat{\theta}\), also called its root mean squared error, is the square root of its pMSE:
\[{\operatorname{pRMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} \stackrel{\text{def}}{=}\sqrt{{\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}}\]
Source: Wikipedia contributors [14].
Example 9 (Population root mean squared error of the sample mean) In Example 8, the pRMSE of \(\bar X\) is \(\sigma/\sqrt{n}\).
Remark 2 (Units of the population root mean squared error). The pRMSE has the same units as the estimand.
Casella and Berger [10, Sec. 7.3.1], pp. 330–333 covers mean squared error.
Definition 10 (Unbiased estimator) An estimator \(\hat{\theta}\) is unbiased if its bias is zero:
\[ {\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = 0 \]
Definition 11 (Biased estimator) An estimator \(\hat{\theta}\) is biased if its bias is not zero:
\[ {\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} \neq 0 \]
Example 10 (The sample mean is unbiased) By Example 7, \({\operatorname{Bias}\mathopen{}\left(\bar X\right)\mathclose{}} = 0\) and \({\operatorname{Bias}\mathopen{}\left(X_1\right)\mathclose{}} = 0\), so both the sample mean \(\bar X\) and the single-observation estimator \(X_1\) are unbiased estimators of \(\mu\).
Example 11 (A biased estimator of the variance) Let \(X_1, \ldots, X_n\) be mutually independent, each with expectation \(\mu\) and variance \(\sigma^2\), and let \(n \ge 2\). Consider two estimators of \(\sigma^2\):
Both estimators are built from the sum of squared deviations \(\sum_{i=1}^n(X_i- \bar X)^2\), so we first find its expectation. Using
\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[X_i^2\right]\mathclose{}} &= {\operatorname{Var}\mathopen{}\left(X_i\right)\mathclose{}} + {\mathopen{}\left({\mathbb{E}\mathopen{}\left[X_i\right]\mathclose{}}\right)\mathclose{}}^{2} \\ &= \sigma^2+ \mu^2 \end{aligned} \]
and
\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[\bar X^2\right]\mathclose{}} &= {\operatorname{Var}\mathopen{}\left(\bar X\right)\mathclose{}} + {\mathopen{}\left({\mathbb{E}\mathopen{}\left[\bar X\right]\mathclose{}}\right)\mathclose{}}^{2} \\ &= \sigma^2/n + \mu^2 \end{aligned} \]
\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[\sum_{i=1}^n(X_i- \bar X)^2\right]\mathclose{}} &= {\mathbb{E}\mathopen{}\left[\sum_{i=1}^n\mathopen{}\left(X_i^2 - 2 X_i\bar X + \bar X^2\right)\mathclose{}\right]\mathclose{}} && \text{(expand each square)}\\ &= {\mathbb{E}\mathopen{}\left[\sum_{i=1}^nX_i^2 - \sum_{i=1}^n2 X_i\bar X + \sum_{i=1}^n\bar X^2\right]\mathclose{}} && \text{(distribute the sum over the terms)}\\ &= {\mathbb{E}\mathopen{}\left[\sum_{i=1}^nX_i^2 - 2\bar X \sum_{i=1}^nX_i+ \sum_{i=1}^n\bar X^2\right]\mathclose{}} && \text{(factor $2\bar X$ out of the middle sum)}\\ &= {\mathbb{E}\mathopen{}\left[\sum_{i=1}^nX_i^2 - 2\bar X \sum_{i=1}^nX_i+ n \bar X^2\right]\mathclose{}} && \text{(sum of $n$ copies of $\bar X^2$)}\\ &= {\mathbb{E}\mathopen{}\left[\sum_{i=1}^nX_i^2 - 2\bar X \cdot n \bar X + n \bar X^2\right]\mathclose{}} && \text{($\textstyle\sum_{i=1}^nX_i= n \bar X$)}\\ &= {\mathbb{E}\mathopen{}\left[\sum_{i=1}^nX_i^2 - 2n \bar X^2 + n \bar X^2\right]\mathclose{}} && \text{(multiply $2\bar X \cdot n \bar X$)}\\ &= {\mathbb{E}\mathopen{}\left[\sum_{i=1}^nX_i^2 - n \bar X^2\right]\mathclose{}} && \text{(collect the $\bar X^2$ terms)}\\ &= {\mathbb{E}\mathopen{}\left[\sum_{i=1}^nX_i^2\right]\mathclose{}} - {\mathbb{E}\mathopen{}\left[n \bar X^2\right]\mathclose{}} && \text{(linearity of expectation)}\\ &= \sum_{i=1}^n{\mathbb{E}\mathopen{}\left[X_i^2\right]\mathclose{}} - {\mathbb{E}\mathopen{}\left[n \bar X^2\right]\mathclose{}} && \text{(linearity of expectation)}\\ &= \sum_{i=1}^n{\mathbb{E}\mathopen{}\left[X_i^2\right]\mathclose{}} - n {\mathbb{E}\mathopen{}\left[\bar X^2\right]\mathclose{}} && \text{(factor the constant $n$ out of the expectation)}\\ &= \sum_{i=1}^n(\sigma^2+ \mu^2) - n {\mathbb{E}\mathopen{}\left[\bar X^2\right]\mathclose{}} && \text{(substitute ${\mathbb{E}\mathopen{}\left[X_i^2\right]\mathclose{}}$)}\\ &= n(\sigma^2+ \mu^2) - n {\mathbb{E}\mathopen{}\left[\bar X^2\right]\mathclose{}} && \text{(sum of $n$ copies of $\sigma^2+ \mu^2$)}\\ &= n(\sigma^2+ \mu^2) - n\mathopen{}\left(\frac{\sigma^2}{n} + \mu^2\right)\mathclose{} && \text{(substitute ${\mathbb{E}\mathopen{}\left[\bar X^2\right]\mathclose{}}$)}\\ &= n\sigma^2+ n\mu^2 - n\mathopen{}\left(\frac{\sigma^2}{n} + \mu^2\right)\mathclose{} && \text{(distribute $n$ over $\sigma^2+ \mu^2$)}\\ &= n\sigma^2+ n\mu^2 - \mathopen{}\left(n \cdot \frac{\sigma^2}{n} + n\mu^2\right)\mathclose{} && \text{(distribute $n$ over $\frac{\sigma^2}{n} + \mu^2$)}\\ &= n\sigma^2+ n\mu^2 - \mathopen{}\left(\sigma^2+ n\mu^2\right)\mathclose{} && \text{(cancel $n$ in $n \cdot \frac{\sigma^2}{n}$)}\\ &= n\sigma^2+ n\mu^2 - \sigma^2- n\mu^2 && \text{(distribute the minus sign)}\\ &= n\sigma^2- \sigma^2 && \text{(cancel $n\mu^2$)}\\ &= (n - 1)\sigma^2 && \text{(factor out $\sigma^2$)} \end{aligned} \]
So
\[ \begin{aligned} {\mathbb{E}\mathopen{}\left[S^2\right]\mathclose{}} &= \frac{1}{n-1}(n-1)\sigma^2\\ &= \sigma^2, \end{aligned} \]
and by Theorem 1, \({\operatorname{Bias}\mathopen{}\left(S^2\right)\mathclose{}} = 0\): the sample variance is unbiased.
For the divide-by-\(n\) estimator, \({\mathbb{E}\mathopen{}\left[\hat{\sigma}^2\right]\mathclose{}} = \frac{1}{n}(n-1)\sigma^2\), so:
\[ \begin{aligned} {\operatorname{Bias}\mathopen{}\left(\hat{\sigma}^2\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[\hat{\sigma}^2\right]\mathclose{}} - \sigma^2 && \text{(bias equals expectation minus truth)}\\ &= \frac{n-1}{n}\sigma^2- \sigma^2 && \text{(substitute ${\mathbb{E}\mathopen{}\left[\hat{\sigma}^2\right]\mathclose{}}$)}\\ &= \frac{n-1}{n}\sigma^2- \frac{n}{n}\sigma^2 && \text{(write $\sigma^2$ over the common denominator $n$)}\\ &= \frac{n - 1 - n}{n}\sigma^2 && \text{(combine the fractions)}\\ &= \frac{-1}{n}\sigma^2 && \text{($n - 1 - n = -1$)}\\ &= -\frac{\sigma^2}{n} && \text{(multiply)} \end{aligned} \]
This estimator is biased: on average it underestimates \(\sigma^2\), by an amount that shrinks to zero as \(n\) grows.
Solution 5. \[ \begin{aligned} 0 &= {\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} && \text{(definition of unbiased)}\\ &= {\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}} - \theta && \text{(bias equals expectation minus truth)} \end{aligned} \]
Adding \(\theta\) to both sides gives \({\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}} = \theta\).
Solution 6. \[ \begin{aligned} {\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} &= {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} + {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} && \text{(MSE equals bias squared plus variance)}\\ &= 0^2 + {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} && \text{(definition of unbiased)}\\ &= {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} && \text{($0^2 = 0$)} \end{aligned} \]
Theorem 6 (Properties of unbiased estimators) If \(\hat{\theta}\) is an unbiased estimator of \(\theta\), then:
\[{\mathbb{E}\mathopen{}\left[\hat{\theta}\right]\mathclose{}} = \theta \tag{5}\]
\[{\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} \tag{6}\]
Proof. Equation 5 is the solution to Exercise 5, and Equation 6 is the solution to Exercise 6.
5MinuteAI covers the properties of estimators, including bias, following the University of Washington’s CSE 312 [15].
Definition 12 (Mean absolute error) The population mean absolute error (pMAE) of an estimator, also called its mean absolute error, is the expectation of the absolute value of the estimation error:
\[ {\operatorname{MAE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} \stackrel{\text{def}}{=}{\mathbb{E}\mathopen{}\left[\mathopen{}\left|{\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right|\mathclose{}\right]\mathclose{}} \]
Source: Wikipedia contributors [16].
Example 12 (Population mean absolute error of a Gaussian estimator) Suppose an estimator \(\hat{\theta}\) has a Gaussian distribution with mean \(\theta\) and standard deviation \(\tau\), so that its estimation error \({\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = \hat{\theta}- \theta\) has the same distribution as \(\tau Z\), where \(Z\) has a standard Gaussian distribution with density \(\phi(z) = (2\pi)^{-1/2} e^{-z^2/2}\). Then:
\[ \begin{aligned} {\operatorname{MAE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} &= {\mathbb{E}\mathopen{}\left[\mathopen{}\left|\tau Z\right|\mathclose{}\right]\mathclose{}} && \text{(definition of MAE)}\\ &= {\mathbb{E}\mathopen{}\left[\tau\mathopen{}\left|Z\right|\mathclose{}\right]\mathclose{}} && \text{($\mathopen{}\left|\tau Z\right|\mathclose{} = \tau\mathopen{}\left|Z\right|\mathclose{}$ because $\tau> 0$)}\\ &= \tau{\mathbb{E}\mathopen{}\left[\mathopen{}\left|Z\right|\mathclose{}\right]\mathclose{}} && \text{(factor the constant $\tau$ out of the expectation)}\\ &= \tau\int_{-\infty}^{\infty} \mathopen{}\left|z\right|\mathclose{} \phi(z) \, dz && \text{(expectation of a function of $Z$)}\\ &= \tau\cdot 2 \int_{0}^{\infty} \mathopen{}\left|z\right|\mathclose{} \phi(z) \, dz && \text{($\mathopen{}\left|z\right|\mathclose{}\phi(z)$ is symmetric about 0)}\\ &= \tau\cdot 2 \int_{0}^{\infty} z \phi(z) \, dz && \text{($\mathopen{}\left|z\right|\mathclose{} = z$ for $z \ge 0$)}\\ &= \tau\cdot 2 \mathopen{}\left[-\phi(z)\right]\mathclose{}_{0}^{\infty} && \text{($\phi'(z) = -z\phi(z)$)}\\ &= \tau\cdot 2 \mathopen{}\left(\lim_{z \to \infty} \mathopen{}\left[-\phi(z)\right]\mathclose{} - \mathopen{}\left(-\phi(0)\right)\mathclose{}\right)\mathclose{} && \text{(evaluate at the limits)}\\ &= \tau\cdot 2 \mathopen{}\left(\lim_{z \to \infty} \mathopen{}\left[-\phi(z)\right]\mathclose{} + \phi(0)\right)\mathclose{} && \text{(subtracting a negative is adding)}\\ &= \tau\cdot 2 \mathopen{}\left(0 + \phi(0)\right)\mathclose{} && \text{($\phi(z) \to 0$ as $z \to \infty$)}\\ &= \tau\cdot 2 \phi(0) && \text{(drop the zero term)}\\ &= \tau\cdot 2 (2\pi)^{-1/2} && \text{($\phi(0) = (2\pi)^{-1/2}$)}\\ &= \tau\cdot 2 \cdot \frac{1}{\sqrt{2\pi}} && \text{(write $(2\pi)^{-1/2}$ as $1/\sqrt{2\pi}$)}\\ &= \tau\cdot \frac{2}{\sqrt{2\pi}} && \text{(multiply)}\\ &= \tau\cdot \frac{\sqrt{4}}{\sqrt{2\pi}} && \text{(write $2$ as $\sqrt{4}$)}\\ &= \tau\sqrt{4/(2\pi)} && \text{(quotient of square roots)}\\ &= \tau\sqrt{2/\pi} && \text{(cancel the common factor 2 in $4/(2\pi)$)} \end{aligned} \]
So for an unbiased Gaussian estimator, the pMAE is about \(0.80\) times the standard deviation \(\tau\), while the pRMSE is \(\tau\) itself (Theorem 6).
Definition 13 (Standard error) The standard error of an estimator \(\hat{\theta}\) is the standard deviation of \(\hat{\theta}\):
\[{\operatorname{SE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} \stackrel{\text{def}}{=}{\operatorname{SD}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\]
Source: Wikipedia contributors [17].
Example 13 (Standard error of the sample mean) In Example 8, \({\operatorname{Var}\mathopen{}\left(\bar X\right)\mathclose{}} = \sigma^2/ n\), so
\[ \begin{aligned} {\operatorname{SE}\mathopen{}\left(\bar X\right)\mathclose{}} &= \sqrt{\sigma^2 / n} \\ &= \sigma/ \sqrt{n}. \end{aligned} \]
With \(n = 50\) students, the standard error of the sample mean height is \(\sigma/ \sqrt{50} \approx 0.14\sigma\).
Exercise 7 (Spread of the estimation error) Find the standard deviation of the estimation error \({\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\) in terms of the standard error of \(\hat{\theta}\).
Solution 7. \[ \begin{aligned} {\operatorname{Var}\mathopen{}\left({\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}} &= {\operatorname{Var}\mathopen{}\left(\hat{\theta}- \theta\right)\mathclose{}} && \text{(definition of estimation error)}\\ &= {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} && \text{(subtracting a constant does not change a variance)} \end{aligned} \]
Taking square roots of both sides,
\[ \begin{aligned} {\operatorname{SD}\mathopen{}\left({\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}} &= {\operatorname{SD}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} \\ &= {\operatorname{SE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}. \end{aligned} \]
Theorem 7 (Standard error is the spread of the estimation error) \[{\operatorname{SE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = {\operatorname{SD}\mathopen{}\left({\varepsilon\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}\]
Proof. This is the solution to Exercise 7.
Remark 3 (The name “standard error”). “Standard error” is a confusing name in two ways. It is defined through the estimator’s own spread, not through the estimation error (although Theorem 7 shows that the two spreads are equal). It is also a synonym for the standard deviation of an estimator, so it can look redundant. The name persists because standard errors are the building blocks of p-values and confidence intervals, so they come up often enough to deserve their own name.
Exercise 8 (Squared standard error from pMSE and bias) Using Theorem 5, write the squared standard error \({\mathopen{}\left({\operatorname{SE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2}\) in terms of the pMSE and the bias of \(\hat{\theta}\).
Solution 8. \[ \begin{aligned} {\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} &= {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} + {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} && \text{(MSE equals bias squared plus variance)}\\ {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} &= {\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} - {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} && \text{(subtract ${\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2}$ from both sides)}\\ {\mathopen{}\left({\operatorname{SD}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} &= {\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} - {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} && \text{(${\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = {\mathopen{}\left({\operatorname{SD}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2}$)}\\ {\mathopen{}\left({\operatorname{SE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} &= {\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} - {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} && \text{(definition of standard error: ${\operatorname{SE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = {\operatorname{SD}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}$)} \end{aligned} \]
Corollary 1 (Standard error squared equals pMSE minus squared bias) The squared standard error is what remains of the pMSE after the squared bias is removed:
\[{\mathopen{}\left({\operatorname{SE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2} = {\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} - {\mathopen{}\left({\operatorname{Bias}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\right)\mathclose{}}^{2}\]
Proof. This is the solution to Exercise 8.
Exercise 9 (Standard error of an unbiased estimator) Using Equation 6, write the standard error of an unbiased estimator \(\hat{\theta}\) in terms of its pMSE.
Solution 9. By Equation 6, \({\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = {\operatorname{Var}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}\). Taking square roots of both sides,
\[ \begin{aligned} \sqrt{{\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}} &= {\operatorname{SD}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} \\ &= {\operatorname{SE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}. \end{aligned} \]
Corollary 2 (For unbiased estimators, SE equals pRMSE) If \(\hat{\theta}\) is unbiased, then its standard error equals its pRMSE:
\[{\operatorname{SE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}} = \sqrt{{\operatorname{pMSE}\mathopen{}\left(\hat{\theta}\right)\mathclose{}}}\]
Proof. This is the solution to Exercise 9.