Definition 1 (Statistical inference)Statistical inference is the process of analyzing data to learn about the shape and structure of the probability distribution that generated those data.
Example 1 (Inference about cholesterol in the WCGS population) The WCGS measured total cholesterol in 3,142 of its 3,154 participants. Using those measurements to learn about the distribution of cholesterol among men like the participants, for example its mean, and quantifying how uncertain that knowledge is, is statistical inference.
Statistical inference typically consists of two steps:
fitting a statistical model to data;
summarizing our uncertainty about the parameters of the fitted model, based on the data (and, in Bayesian inference, on prior beliefs).
There are two predominant paradigms for statistical inference:
frequentist inference, which treats parameters as fixed unknown constants;
Bayesian inference, which treats parameters as random variables with prior distributions.
2 Hypothesis tests
2.1 Hypotheses
Definition 2 (Null hypothesis) The null hypothesis, written \(H_0\), is a specific claim about the parameters of a statistical model, which a hypothesis test assesses against the data.
Definition 3 (Alternative hypothesis) The alternative hypothesis, written \(H_1\) or \(H_A\), is the claim that a hypothesis test weighs the data as evidence for, against the null hypothesis.
Definition 4 (Two-sided alternative) For a null hypothesis that a parameter equals a value, such as \(H_0: \theta = \theta_0\), the two-sided alternative is that the parameter differs from that value in either direction: \(H_1: \theta \ne \theta_0\).
Example 2 (Cholesterol and CHD in the WCGS) Let \(\mu_1\) and \(\mu_0\) be the mean cholesterol of men who do and do not have a CHD event by 1969, in the population the WCGS sampled. A comparison of the two groups tests the null hypothesis that the means are equal against the two-sided alternative that they differ:
Definition 5 (Test statistic) A test statistic\(T\) is a function of the data whose distribution under the null hypothesis is known, exactly or approximately, and whose larger values are more extreme, meaning less consistent with \(H_0\) and more consistent with \(H_1\).
Definition 6 (p-value) Given a test statistic\(T\) whose observed value is \(t\), the p-value is the probability that \(T\) is at least as extreme as the value observed, computed with the data distributed as the null hypothesis specifies:
\[p \stackrel{\text{def}}{=}\Pr_{H_0}(T \ge t)\]
If the null hypothesis allows more than one distribution (a composite null hypothesis, such as \(H_0: \mu_1 = \mu_0\) with unknown variances), the p-value is the largest such probability over those distributions, or an approximation to it.
A p-value is computed assuming \(H_0\) is true, so it is not the probability that \(H_0\) is true given the data.
Definition 7 (Significance level) A hypothesis test with significance level\(\alpha\) rejects the null hypothesis when the p-value is at most \(\alpha\).
\(\alpha = 0.05\) is a common convention, not a rule. When \(H_0\) is true and the p-value is computed exactly, a test with significance level \(\alpha\) rejects \(H_0\) with probability at most \(\alpha\); for a test based on an approximate p-value, such as a large-sample test, this holds only approximately.
Definition 8 (Statistically significant) A result is statistically significant at level \(\alpha\) if it leads a test with significance level\(\alpha\) to reject the null hypothesis.
Example 3 (Testing whether mean cholesterol differs by CHD status) Continuing Example 2, let \(\bar x_1\) and \(\bar x_0\) be the two groups’ sample mean cholesterol, with sample variances \(s_1^2\) and \(s_0^2\) and sample sizes \(n_1\) and \(n_0\). By the central limit theorem, when \(H_0\) holds and the samples are large, the statistic
has approximately a standard Gaussian distribution. Extreme values in either direction are evidence for the two-sided \(H_1\), so the test statistic is \(T = \mathopen{}\left|Z\right|\mathclose{}\), and the p-value is \(\Pr(\mathopen{}\left|Z\right|\mathclose{} \ge \mathopen{}\left|z\right|\mathclose{}) = 2\Phi(-\mathopen{}\left|z\right|\mathclose{})\), where \(\Phi\) is the standard Gaussian CDF.
chol_by_chd<-split(wcgs$chol, wcgs$chd69)|>lapply(function(x)x[!is.na(x)])x1<-chol_by_chd[["Yes"]]x0<-chol_by_chd[["No"]]diff_means<-mean(x1)-mean(x0)se_diff<-sqrt(var(x1)/length(x1)+var(x0)/length(x0))z<-diff_means/se_diffp_value<-2*pnorm(-abs(z))c(difference =diff_means, se =se_diff, z =z, p_value =p_value)#> difference se z p_value #> 2.58087e+01 3.17988e+00 8.11626e+00 4.80780e-16
The p-value is far below 0.05, so at significance level 0.05 the test rejects \(H_0\): the data provide statistically significant evidence that mean cholesterol differs between the two groups.
3 Reference distributions
The tests on this page compare a test statistic with one of three families of distributions, each built from independent standard Gaussian random variables. Here, “independent” means mutually independent.
Definition 9 (Chi-square distribution) Let \(Z_1, \ldots, Z_k\) be independent random variables, each with the standard Gaussian distribution. The distribution of \(\sum_{j=1}^k Z_j^2\) is the chi-square distribution with \(k\) degrees of freedom, written \(\chi^2_k\).
Example 4 (The 0.95 quantile of \(\chi^2_1\)) By Definition 9, a \(\chi^2_1\) random variable is \(Z^2\) for a standard Gaussian \(Z\). So \(\Pr(Z^2 \le c) = \Pr(-\sqrt{c} \le Z \le \sqrt{c})\), and the 0.95 quantile of \(\chi^2_1\) is the square of the 0.975 quantile of the standard Gaussian distribution:
Definition 10 (t-distribution) Let \(Z\) have the standard Gaussian distribution, let \(V\) have the \(\chi^2_k\) distribution (Definition 9), and let \(Z\) and \(V\) be independent. The distribution of
\[\frac{Z}{\sqrt{V/k}}\]
is the t-distribution with \(k\) degrees of freedom (or Student’s t-distribution), written \(t_k\).
Example 5 (t quantiles approach Gaussian quantiles) The 0.975 quantile of \(t_k\) is larger than the standard Gaussian’s 0.975 quantile, 1.96, and approaches it as \(k\) grows:
So t-based intervals and tests differ noticeably from Gaussian-based ones only in small samples.
Definition 11 (F-distribution) Let \(U\) have the \(\chi^2_{k_1}\) distribution, let \(V\) have the \(\chi^2_{k_2}\) distribution, and let \(U\) and \(V\) be independent. The distribution of
\[\frac{U / k_1}{V / k_2}\]
is the F-distribution with \(k_1\) and \(k_2\) degrees of freedom, written \(F_{k_1, k_2}\).
Example 6 (A squared t random variable has an F-distribution) Let \(T = Z / \sqrt{V/k}\) as in Definition 10. Then
\[T^2 = \frac{Z^2 / 1}{V / k},\]
where \(Z^2\) has the \(\chi^2_1\) distribution (Definition 9) and is independent of \(V\). So \(T^2\) has the \(F_{1, k}\) distribution (Definition 11), and the 0.95 quantile of \(F_{1, k}\) is the square of the 0.975 quantile of \(t_k\):
Definition 12 (Confidence interval) A \(100(1-\alpha)\%\)confidence interval for a parameter \(\theta\) is a pair of statistics \(L\) and \(U\), computed from the data, such that for every possible value of \(\theta\):
\[\Pr(L \le \theta \le U) = 1 - \alpha\]
The probability is over repeated samples from the data-generating process, with \(\theta\) held fixed.
Definition 13 (Coverage probability) The coverage probability of an interval \([L, U]\) for a parameter \(\theta\) is \(\Pr(L \le \theta \le U)\), computed over repeated samples with \(\theta\) held fixed. A \(100(1-\alpha)\%\)confidence interval has coverage probability \(1 - \alpha\).
Definition 14 (Approximate confidence interval) An approximate\(100(1-\alpha)\%\) confidence interval is an interval whose coverage probability is approximately \(1 - \alpha\), for example only in large samples.
Example 7 (Confidence interval for mean cholesterol in the WCGS) By the central limit theorem, the sample mean \(\bar X\) of a large sample has approximately a Gaussian distribution with mean \(\mu\) and standard error\(\sigma/\sqrt{n}\), which we estimate by \(s/\sqrt{n}\). So \(\bar x \pm z_{0.975} \, s/\sqrt{n}\), where \(z_{0.975} \approx 1.96\) is the 0.975 quantile of the standard Gaussian distribution, is an approximate 95% confidence interval for \(\mu\):
If a confidence interval includes the null value, or a hypothesis test fails to reject the null hypothesis, that result does not necessarily mean that the null hypothesis is true. (When the interval and the test are built from the same statistic, with coverage \(1 - \alpha\) and significance level \(\alpha\), the two results coincide.) So we should not interpret such results as “the odds (or risks, hazards, or means) are not significantly different”. Instead, we should write something like “the data do not provide statistically significant evidence that the odds (or risks, hazards, or means) differ”. Statistical significance is a property of evidence, not of the estimands.
P-values do not distinguish between absence of evidence and evidence of absence. Confidence intervals do:
a narrow confidence interval that includes the null value is evidence of absence (of any effect large enough to matter);
a confidence interval that includes the null value but also includes substantially non-null values represents absence of evidence.
Conversely, even statistically significant evidence of a non-null value does not mean the value is large enough to matter; that depends on what the estimand is. For example, we might have statistically significant evidence that an exercise program prolongs human lifespans by 20 seconds, but an effect that small would be negligible in practical terms.
ci_scenarios<-tibble::tribble(~scenario, ~lower, ~upper,"not enough data: we know almost nothing", 0.2, 5,"absence of evidence", 0.5, 2,"\"trending toward significance\" (still absence of evidence)", 0.95, 2.5,"evidence of absence", 0.97, 1.03,"evidence of a negligible effect", 1.02, 1.1,"uncertain whether negligible or substantial", 1.05, 1.8,"evidence of a substantial effect", 1.7, 1.8,"substantial, but imprecise", 1.5, 4)|>dplyr::mutate(scenario =factor(scenario, levels =rev(scenario)))ci_scenarios|>ggplot2::ggplot()+ggplot2::aes(y =scenario, xmin =lower, xmax =upper)+ggplot2::geom_errorbarh(height =0.3)+ggplot2::geom_vline(xintercept =1, linetype ="dashed")+ggplot2::scale_x_log10()+ggplot2::labs(x ="Ratio estimand (log scale); null value = 1", y =NULL)
Figure 1: Interpretations of hypothetical 95% confidence intervals for a ratio (such as an odds ratio), relative to the null value 1 (dashed line). Adapted from a whiteboard sketch made in office hours.
Vittinghoff, Eric, David V Glidden, Stephen C Shiboski, and Charles E McCulloch. 2012. Regression Methods in Biostatistics: Linear, Logistic, Survival, and Repeated Measures Models. 2nd ed. Springer. https://doi.org/10.1007/978-1-4614-1353-0.
---title: "Statistical Inference"format: html: default revealjs: output-file: inference-slides.html pdf: output-file: inference-handout.pdf docx: output-file: inference-handout.docx---## Inference{{< include sds/_subfiles/shared-config.qmd >}}```{r}#| label: load-wcgs#| include: falsewcgs <- rmb::wcgs |> haven::as_factor()```<!-- No slidebreak here: the heading shares its slide with the def-inference div. -->{{< include sds/_subfiles/inference/_def-inference.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_exm-inference.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_prose-inference-steps.qmd >}}## Hypothesis tests {#sec-hypothesis-tests}### Hypotheses<!-- No slidebreak here: the heading shares its slide with the def-null-hypothesis div. -->{{< include sds/_subfiles/inference/_def-null-hypothesis.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_def-alternative-hypothesis.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_def-two-sided-alternative.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_exm-hypotheses-wcgs.qmd >}}{{< slidebreak >}}### p-values<!-- No slidebreak here: the heading shares its slide with the def-test-statistic div. -->{{< include sds/_subfiles/inference/_def-test-statistic.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_def-p-value.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_def-significance-level.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_def-statistically-significant.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_exm-p-value-wcgs.qmd >}}## Reference distributions {#sec-reference-distributions}{{< include sds/_subfiles/basic-statistical-methods/_prose-reference-distributions.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/basic-statistical-methods/_def-chi-square-dist.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/basic-statistical-methods/_exm-chi-square-dist.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/basic-statistical-methods/_def-t-dist.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/basic-statistical-methods/_exm-t-dist.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/basic-statistical-methods/_def-f-dist.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/basic-statistical-methods/_exm-f-dist.qmd >}}## Confidence intervals {#sec-CI}<!-- No slidebreak here: the heading shares its slide with the def-confidence-interval div. -->{{< include sds/_subfiles/inference/_def-confidence-interval.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_def-coverage-probability.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_def-approximate-ci.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_exm-confidence-interval-wcgs.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_def-margin-error.qmd >}}{{< slidebreak >}}{{< include sds/_subfiles/inference/_exm-margin-error.qmd >}}## Further reading on confidence intervals{{< include sds/_subfiles/inference/_prose-CI-resources.qmd >}}## Interpretation of negative findings{{< include sds/_subfiles/inference/_prose-negative-findings.qmd >}}## References {.unnumbered}::: {#refs}:::
