Statistics

Samples, estimation, inference, and testing

Published

Last modified: 2026-10-18

This appendix outlines the statistical foundations and terminology assumed throughout these notes. It points to the definitions, worked calculations, and examples in the lab’s shared prerequisite site: Statistics for Data Science.

Machine learning and statistics share much of their mathematical foundations, but often differ in their primary goals: statistics emphasizes inference, parameter estimation, and hypothesis testing, whereas machine learning emphasizes generalization and predictive accuracy on unseen data [1], [2].

Statistical theory explains what conclusions a finite sample of data supports, how to estimate underlying model parameters, and how to measure uncertainty. This appendix outlines the core concepts and links directly to their full definitions in Statistics for Data Science.

1 Populations, samples, and empirical distributions

Statistics for Data Science covers the distinction between populations and finite samples:

  • Populations and samples: A population distribution describes the entire data-generating process, while a sample consists of \(N\) observed data points drawn from that population.
  • Independent and identically distributed (i.i.d.) observations: Sampling data independently from a shared underlying distribution \(p(\vx, y)\).
  • Empirical distributions: Placing probability mass \(1/N\) on each observed training instance, which forms the basis for empirical risk minimization [3].
  • Standardization and min-max scaling: Centering features to mean zero and scaling to unit variance (z-scores), or scaling to a fixed interval \([0, 1]\).
VideosWatch
ReadingRead

2 Point estimation and estimator properties

Statistics for Data Science covers parameter estimation and estimator performance:

  • Point estimators: A rule or statistic \(\hparam = g(X_1, \ldots, X_N)\) used to estimate an unknown population parameter \(\param\) [7].
  • Estimator bias: \(\mathrm{Bias}(\hparam) \triangleq \Ep[\hparam] - \param\). An estimator is unbiased when \(\mathrm{Bias}(\hparam) = 0\) [8].
  • Estimator variance and standard error: \(\Var{\hparam} = \Ep\sb{(\hparam - \Ep[\hparam])^2}\), with standard error \(\mathrm{SE}(\hparam) \triangleq \sqrt{\Var{\hparam}}\).
  • Mean squared error (MSE) and the bias-variance decomposition of an estimator: \(\pMSE{\hparam} \triangleq \Ep\sb{(\hparam - \param)^2} = \mathrm{Bias}(\hparam)^2 + \Var{\hparam}\).
  • Maximum likelihood estimation (MLE): Estimating parameters by maximizing the likelihood function \(L(\param; \vx) = \prod_{i=1}^N p(x_i; \param)\), or equivalently the log-likelihood \(\llik(\param; \vx) = \sumiN \ln p(x_i; \param)\).
VideosWatch

5MinuteAI explains:

Both come from its Probability & Statistics playlist [13].

ReadingRead

The University of Washington’s CSE 446 slides cover maximum likelihood estimation [14], Lecture 2.

3 Hypothesis testing and confidence intervals

Statistics for Data Science covers formal hypothesis testing and uncertainty quantification:

  • Null hypothesis (\(H_0\), the default baseline or null effect) versus alternative hypothesis (\(H_1\), the alternative claim).
  • Test statistics and \(p\)-values: A test statistic summarizes sample evidence, and the \(p\)-value is the probability under \(H_0\) of observing a test statistic at least as extreme as the one calculated.
  • Significance levels and decision errors: A test with significance level \(\siglevel\) rejects \(H_0\) when the \(p\)-value is at most \(\siglevel\). A Type I error (false positive) occurs when rejecting a true \(H_0\); a Type II error (false negative) occurs when failing to reject a false \(H_0\), with statistical power defined as \(1 - \typetwoerr\).
  • Confidence intervals: Constructing an interval \([L, U]\) that covers the true parameter \(\param\) with probability \(1 - \siglevel\) under repeated sampling [15].
VideosWatch

5MinuteAI explains confidence intervals and hypothesis testing [13].

ReadingRead

4 Statistical inference versus machine learning prediction

Machine learning and statistics share data and mathematical tools, but often approach modeling from different perspectives:

  • Inference: Focuses on understanding the data-generating mechanism, estimating interpretable parameters, quantifying uncertainty via standard errors and confidence intervals, and testing specific scientific hypotheses.
  • Prediction: Focuses on constructing models that generalize accurately to new, unseen observations, often accepting complex, regularized, or non-parametric models where individual parameters may not have a simple mechanistic interpretation.
  • Course Overview discusses the relationship between machine learning and statistics in detail, and provides a translation table between the vocabularies of both fields.
  • Historical and cultural roots: The divergence in terminology and modeling philosophy is surveyed in classic papers by Breiman [18] on the algorithmic and data modeling cultures, Ripley [19], Table 1.1 on parallel terms across neural networks and statistics, Varian [20] comparing machine learning and econometrics, and Efron and Hastie [21] on computer age statistical inference.

5 Further reading

  • James et al. [1], Chapter 2

  • Hastie et al. [2], Chapters 1 and 2

  • Starmer [22], on statistical testing and estimation

References

[1]
G. James, D. Witten, T. Hastie, and R. Tibshirani, An introduction to statistical learning with applications in R, 2nd ed. Springer, 2021. Available: https://www.statlearning.com/
[2]
T. Hastie, R. Tibshirani, and J. Friedman, The elements of statistical learning: Data mining, inference, and prediction, 2nd ed. Springer, 2009. Available: https://hastie.su.domains/ElemStatLearn/
[3]
Wikipedia contributors, “Empirical distribution function — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Empirical_distribution_function
[4]
J. Starmer, “One-hot, label, target and k-fold target encoding, clearly explained.” StatQuest, video on YouTube, 2023. Available: https://www.youtube.com/watch?v=589nCGeWG1w
[5]
GeeksforGeeks, “Feature engineering - scaling, normalization and standardization.” GeeksforGeeks, tutorial article, 2026. Available: https://www.geeksforgeeks.org/machine-learning/feature-engineering-scaling-normalization-and-standardization/
[6]
GeeksforGeeks, “Exploratory data analysis.” GeeksforGeeks, tutorial article, 2026. Available: https://www.geeksforgeeks.org/data-analysis/what-is-exploratory-data-analysis/
[7]
Wikipedia contributors, “Estimator — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Estimator
[8]
Wikipedia contributors, “Bias of an estimator — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Bias_of_an_estimator
[9]
J. Starmer, “Maximum likelihood, clearly explained.” StatQuest, video on YouTube, 2017. Available: https://www.youtube.com/watch?v=XepXtl9YKwc
[10]
J. Starmer, “Machine learning fundamentals: Bias and variance.” StatQuest, video on YouTube, 2018. Available: https://www.youtube.com/watch?v=EuBBz3bI-aA
[11]
GeeksforGeeks, “Probability density estimation & maximum likelihood estimation.” GeeksforGeeks, tutorial article, 2025. Available: https://www.geeksforgeeks.org/machine-learning/probability-density-estimation-maximum-likelihood-estimation/
[12]
GeeksforGeeks, “Bias-variance trade off - machine learning.” GeeksforGeeks, tutorial article, 2026. Available: https://www.geeksforgeeks.org/machine-learning/ml-bias-variance-trade-off/
[13]
5MinuteAI, “Probability & statistics.” Video playlist on YouTube. Accessed: Oct. 12, 2026. [Online]. Available: https://www.youtube.com/playlist?list=PLeB45KifGiuHesi4PALNZSYZFhViVGQJK
[14]
N. Jaques, “CSE 446/546: Machine learning, lecture slides.” University of Washington, 2026. Accessed: Oct. 12, 2026. [Online]. Available: https://courses.cs.washington.edu/courses/cse446/26sp/schedule/lectures/
[15]
Wikipedia contributors, “Confidence interval — Wikipedia, the free encyclopedia.” 2026. Available: https://en.wikipedia.org/wiki/Confidence_interval
[16]
GeeksforGeeks, “Hypothesis testing.” GeeksforGeeks, tutorial article, 2025. Available: https://www.geeksforgeeks.org/data-science/understanding-hypothesis-testing/
[17]
GeeksforGeeks, “Confidence interval.” GeeksforGeeks, tutorial article, 2025. Available: https://www.geeksforgeeks.org/maths/confidence-interval/
[18]
L. Breiman, “Statistical modeling: The two cultures,” Statistical Science, vol. 16, no. 3, 2001, doi: 10.1214/ss/1009213726.
[19]
B. D. Ripley, Pattern recognition and neural networks. Cambridge: Cambridge University Press, 1996. doi: 10.1017/CBO9780511812651.
[20]
H. R. Varian, “Big data: New tricks for econometrics,” Journal of Economic Perspectives, vol. 28, no. 2, pp. 3–28, 2014, doi: 10.1257/jep.28.2.3.
[21]
B. Efron and T. Hastie, Computer age statistical inference: Algorithms, evidence, and data science. Cambridge: Cambridge University Press, 2016. doi: 10.1017/CBO9781316576533.
[22]
J. Starmer, The StatQuest illustrated guide to statistics: With hands-on examples in Python and R. 2026. Available: https://statquest.org/statquest-store/
Back to top