Correlation and Simple Linear Regression
Last modified: 2026-09-29 00:05:17 (PDT)
This page reviews two ways to relate two continuous variables: correlation coefficients, with tests of whether they differ from zero, and simple linear regression. It uses the \(t\) reference distribution defined on the Statistical Inference page. This page is adapted from Vittinghoff et al. (2012), Chapter 3.
1 The HERS data
The examples on this page use the HERS data, which the Comparing Means page describes. The rmb R package includes the dataset; haven::as_factor() converts its Stata value labels to factors:
2 Correlation
2.1 Testing the Pearson correlation
Definition 1 (Population correlation) The population correlation of two random variables \(X\) and \(Y\) with positive variances is
\[\rho \stackrel{\text{def}}{=}\frac{\operatorname{Cov}\mathopen{}\left(X, Y\right)\mathclose{}}{\sqrt{\operatorname{Var}\mathopen{}\left(X\right)\mathclose{} \operatorname{Var}\mathopen{}\left(Y\right)\mathclose{}}},\]
where \(\operatorname{Cov}\mathopen{}\left(X, Y\right)\mathclose{}\) is their covariance.
Definition 2 (Test of zero correlation) Let \(r\) be the Pearson correlation coefficient of \(n \ge 3\) pairs, with \(\mathopen{}\left|r\right|\mathclose{} < 1\). The t-test of zero correlation of \(H_0: \rho = 0\) against \(H_1: \rho \neq 0\) (Definition 1) uses the statistic
\[t \stackrel{\text{def}}{=}\frac{r \sqrt{n - 2}}{\sqrt{1 - r^2}}.\]
Its p-value is \(\Pr(\mathopen{}\left|T\right|\mathclose{} \ge \mathopen{}\left|t\right|\mathclose{})\), where \(T\) has the \(t_{n-2}\) distribution (t-distribution).
Theorem 1 (Null distribution of the correlation t statistic) Let the pairs \((X_1, Y_1), \ldots, (X_n, Y_n)\) be independent, and let each \(Y_i\), given \(X_1, \ldots, X_n\), be Gaussian with a mean and variance that do not depend on the \(X\) values. Then the statistic \(t\) of Definition 2 has the \(t_{n-2}\) distribution (Hogg et al. 2019, sec. 9.6, pp. 472-473).
Example 1 (Correlation between BMI and fasting glucose in HERS) Figure 1 plots baseline fasting glucose against BMI.
[R code]
The statistic of Definition 2, using the 2758 participants with a BMI measurement:
cor.test() reports the same values, and a confidence interval for \(\rho\):
cor.test(hers$BMI, hers$glucose, method = "pearson")
#>
#> Pearson's product-moment correlation
#>
#> data: hers$BMI and hers$glucose
#> t = 14.88, df = 2756, p-value <2e-16
#> alternative hypothesis: true correlation is not equal to 0
#> 95 percent confidence interval:
#> 0.237760 0.306865
#> sample estimates:
#> cor
#> 0.272664The correlation is positive but modest: glucose tends to be higher at higher BMI, with wide scatter around the line in Figure 1.
2.2 Spearman rank correlation
Definition 3 (Spearman rank correlation) Replace each \(x_i\) by its rank among \(x_1, \ldots, x_n\), and each \(y_i\) by its rank among \(y_1, \ldots, y_n\), giving tied values the average of the ranks they span. The Spearman rank correlation \(r_S\) is the Pearson correlation coefficient of the ranks.
Example 2 (Spearman correlation between BMI and fasting glucose in HERS) The Pearson correlation of the ranks (Definition 3), and cor.test()’s Spearman test (exact = FALSE, because tied values rule out the exact p-value):
hers_bmi <- hers |> dplyr::filter(!is.na(BMI))
cor(rank(hers_bmi$BMI), rank(hers_bmi$glucose))
#> [1] 0.333751
cor.test(hers_bmi$BMI, hers_bmi$glucose, method = "spearman", exact = FALSE)
#>
#> Spearman's rank correlation rho
#>
#> data: hers_bmi$BMI and hers_bmi$glucose
#> S = 2.33e+09, p-value <2e-16
#> alternative hypothesis: true rho is not equal to 0
#> sample estimates:
#> rho
#> 0.333751\(r_S\) is larger than the Pearson \(r\) of Example 1: the association is closer to monotone than to linear.
3 Simple linear regression
3.1 Model specification
Definition 4 (Simple linear regression model) A simple linear regression model relates a continuous outcome \(Y\) to a single predictor \(X\):
\[Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \qquad \varepsilon_1, \ldots, \varepsilon_n \ \sim_{\operatorname{iid}}\ \operatorname{N}\mathopen{}\left(0, \sigma^2\right)\mathclose{}.\]
- \(\beta_0\) is the intercept: the mean of \(Y\) among observations with \(X = 0\).
- \(\beta_1\) is the slope: the difference in the mean of \(Y\) between two groups whose values of \(X\) differ by one unit, \(\beta_1 = \operatorname{E}\mathopen{}\left[Y \mid X = x + 1\right]\mathclose{} - \operatorname{E}\mathopen{}\left[Y \mid X = x\right]\mathclose{}\).
- \(\sigma^2\) is the variance of \(Y\) around its mean at each value of \(X\).
3.2 Ordinary least squares estimation
Definition 5 (Residual sum of squares) For data \((x_1, y_1), \ldots, (x_n, y_n)\), the residual sum of squares of a line with intercept \(b_0\) and slope \(b_1\) is
\[\text{RSS}(b_0, b_1) \stackrel{\text{def}}{=}\sum_{i=1}^n (y_i - b_0 - b_1 x_i)^2.\]
Example 3 (Residual sum of squares of a line through three points) For the points \((0, 1)\), \((1, 2)\), \((2, 2)\) and the line with \(b_0 = 1\) and \(b_1 = 0.5\), the vertical distances from the points to the line are \(1 - 1 = 0\), \(2 - 1.5 = 0.5\), and \(2 - 2 = 0\), so \(\text{RSS}(1, 0.5) = 0^2 + 0.5^2 + 0^2 = 0.25\).
Definition 6 (Ordinary least squares) The ordinary least squares (OLS) estimates \(\hat\beta_0\) and \(\hat\beta_1\) are the values of \(b_0\) and \(b_1\) that minimize the residual sum of squares \(\text{RSS}(b_0, b_1)\).
Theorem 2 (Closed-form OLS estimates) Write
\[ \begin{aligned} S_{xx} &\stackrel{\text{def}}{=}\sum_{i=1}^n (x_i - \bar{x})^2, & S_{yy} &\stackrel{\text{def}}{=}\sum_{i=1}^n (y_i - \bar{y})^2, & S_{xy} &\stackrel{\text{def}}{=}\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}), \end{aligned} \]
and suppose \(S_{xx} > 0\) (not all \(x_i\) are equal). Then the OLS estimates (Definition 6) are unique, and
\[\hat\beta_1 = \frac{S_{xy}}{S_{xx}}, \qquad \hat\beta_0 = \bar{y} - \hat\beta_1 \bar{x}.\]
Proof. \(\text{RSS}\) is a quadratic function of \((b_0, b_1)\), so we find where its partial derivatives are zero, then check that this point is the unique minimum.
The derivative with respect to \(b_0\):
\[ \begin{aligned} \frac{\partial \text{RSS}}{\partial b_0} &= -2 \sum_{i=1}^n (y_i - b_0 - b_1 x_i) && \text{(chain rule, term by term)}\\ &= -2 \mathopen{}\left(n \bar{y} - n b_0 - b_1 n \bar{x}\right)\mathclose{} && \text{($\sum_i y_i = n\bar{y}$, $\sum_i x_i = n\bar{x}$)} \end{aligned} \]
Setting it to zero and dividing by \(-2n\) gives \(\bar{y} - b_0 - b_1 \bar{x} = 0\), so \(b_0 = \bar{y} - b_1 \bar{x}\).
The derivative with respect to \(b_1\), after substituting \(b_0 = \bar{y} - b_1 \bar{x}\):
\[ \begin{aligned} \frac{\partial \text{RSS}}{\partial b_1} &= -2 \sum_{i=1}^n x_i (y_i - b_0 - b_1 x_i) && \text{(chain rule, term by term)}\\ &= -2 \sum_{i=1}^n x_i \mathopen{}\left((y_i - \bar{y}) - b_1 (x_i - \bar{x})\right)\mathclose{} && \text{(substitute $b_0$)}\\ &= -2 \sum_{i=1}^n (x_i - \bar{x}) \mathopen{}\left((y_i - \bar{y}) - b_1 (x_i - \bar{x})\right)\mathclose{} && \text{(subtract $\bar{x} \sum_i \mathopen{}\left((y_i - \bar{y}) - b_1 (x_i - \bar{x})\right)\mathclose{} = 0$)}\\ &= -2 \mathopen{}\left(S_{xy} - b_1 S_{xx}\right)\mathclose{} && \text{(definitions of $S_{xy}$ and $S_{xx}$)} \end{aligned} \]
The subtracted sum in the third step is zero because \(\sum_i (y_i - \bar{y}) = 0\) and \(\sum_i (x_i - \bar{x}) = 0\). Setting the derivative to zero gives \(b_1 = S_{xy} / S_{xx}\).
This stationary point is the unique minimum. The matrix of second derivatives of \(\text{RSS}\) is
\[ 2 \begin{pmatrix} n & n\bar{x} \\ n\bar{x} & \sum_i x_i^2 \end{pmatrix}, \]
whose top-left entry \(2n\) is positive and whose determinant is \(4\mathopen{}\left(n \sum_i x_i^2 - n^2 \bar{x}^2\right)\mathclose{} = 4 n S_{xx} > 0\). So the matrix is positive definite, \(\text{RSS}\) is strictly convex, and its only stationary point is its global minimum.
Corollary 1 (OLS slope in terms of the correlation) If \(S_{xx} > 0\) and \(S_{yy} > 0\), then
\[\hat\beta_1 = r \, \frac{s_y}{s_x},\]
where \(r\) is the Pearson correlation coefficient and \(s_x\) and \(s_y\) are the sample standard deviations of the \(x_i\) and the \(y_i\).
Proof. With the notation of Theorem 2, \(r = S_{xy} / \sqrt{S_{xx} S_{yy}}\), \(s_x = \sqrt{S_{xx} / (n-1)}\), and \(s_y = \sqrt{S_{yy} / (n-1)}\). So:
\[ \begin{aligned} r \, \frac{s_y}{s_x} &= \frac{S_{xy}}{\sqrt{S_{xx} S_{yy}}} \cdot \frac{\sqrt{S_{yy} / (n-1)}}{\sqrt{S_{xx} / (n-1)}} && \text{(substitute $r$, $s_x$, $s_y$)}\\ &= \frac{S_{xy}}{\sqrt{S_{xx}} \sqrt{S_{yy}}} \cdot \frac{\sqrt{S_{yy}}}{\sqrt{S_{xx}}} && \text{(cancel the factors of $n - 1$)}\\ &= \frac{S_{xy}}{S_{xx}} && \text{(cancel $\sqrt{S_{yy}}$)}\\ &= \hat\beta_1 && \text{(closed-form OLS slope)} \end{aligned} \]
Video lecture
Hutchinson’s Linear Regression (pt1) (17 min) covers fitting a linear regression by least squares, framed as machine learning (Hutchinson, n.d.). The login for the video site is posted on Canvas.
3.3 Fitting a simple linear regression in R
Example 4 (Regression of fasting glucose on BMI in HERS) The OLS estimates from Theorem 2, and the slope from Corollary 1, for the participants with a BMI measurement:
hers_bmi <- hers |> dplyr::filter(!is.na(BMI))
x <- hers_bmi$BMI
y <- hers_bmi$glucose
b1 <- sum((x - mean(x)) * (y - mean(y))) / sum((x - mean(x))^2)
c(
intercept = mean(y) - b1 * mean(x),
slope = b1,
slope_from_r = cor(x, y) * sd(y) / sd(x)
)
#> intercept slope slope_from_r
#> 60.07405 1.82218 1.82218lm() gives the same estimates:
slr_fit <- lm(glucose ~ BMI, data = hers)
summary(slr_fit)
#>
#> Call:
#> lm(formula = glucose ~ BMI, data = hers)
#>
#> Residuals:
#> Min 1Q Median 3Q Max
#> -81.55 -18.98 -10.35 3.76 190.81
#>
#> Coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) 60.074 3.565 16.9 <2e-16 ***
#> BMI 1.822 0.122 14.9 <2e-16 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 35.5 on 2756 degrees of freedom
#> (5 observations deleted due to missingness)
#> Multiple R-squared: 0.0743, Adjusted R-squared: 0.074
#> F-statistic: 221 on 1 and 2756 DF, p-value: <2e-16The estimated slope is \(\hat\beta_1 = 1.82\) mg/dL per kg/m2: mean fasting glucose is about 1.8 mg/dL higher among participants whose BMI is 1 kg/m2 higher. The t statistic for the slope equals the correlation test statistic of Example 1.
3.4 The coefficient of determination
Definition 7 (Total sum of squares) The total sum of squares of \(y_1, \ldots, y_n\) is
\[\text{TSS} \stackrel{\text{def}}{=}\sum_{i=1}^n (y_i - \bar{y})^2.\]
Example 5 (Total sum of squares of three values) For \(y = 1, 2, 2\), \(\bar y = 5/3\), so
\[ \begin{aligned} \text{TSS} &= \mathopen{}\left(1 - \tfrac{5}{3}\right)\mathclose{}^2 + 2\mathopen{}\left(2 - \tfrac{5}{3}\right)\mathclose{}^2 && \text{(definition)}\\ &= \tfrac{4}{9} + \tfrac{2}{9} && \text{(square the deviations)}\\ &= \tfrac{2}{3} && \text{(arithmetic)} \end{aligned} \]
Definition 8 (Coefficient of determination) For a fitted regression with fitted values \(\hat{y}_i\), the coefficient of determination is
\[R^2 \stackrel{\text{def}}{=}1 - \frac{\sum_{i=1}^n (y_i - \hat{y}_i)^2}{\sum_{i=1}^n (y_i - \bar{y})^2}.\]
The numerator is the residual sum of squares of the fit, and the denominator is the total sum of squares of the \(y_i\).
Theorem 3 (\(R^2\) of a simple linear regression) For the OLS fit of a simple linear regression, with \(S_{xx} > 0\) and \(S_{yy} > 0\), \(R^2 = r^2\), where \(r\) is the Pearson correlation coefficient of the \(x_i\) and \(y_i\). In particular, \(0 \le R^2 \le 1\).
Proof. With the notation of Theorem 2, the fitted values are \(\hat{y}_i = \hat\beta_0 + \hat\beta_1 x_i\), so each residual is
\[ \begin{aligned} y_i - \hat{y}_i &= y_i - (\bar{y} - \hat\beta_1 \bar{x}) - \hat\beta_1 x_i && \text{(substitute $\hat\beta_0$)}\\ &= (y_i - \bar{y}) - \hat\beta_1 (x_i - \bar{x}) && \text{(regroup)} \end{aligned} \]
Squaring and summing:
\[ \begin{aligned} \sum_{i=1}^n (y_i - \hat{y}_i)^2 &= \sum_{i=1}^n \mathopen{}\left((y_i - \bar{y})^2 - 2 \hat\beta_1 (x_i - \bar{x})(y_i - \bar{y}) + \hat\beta_1^2 (x_i - \bar{x})^2\right)\mathclose{} && \text{(expand the square)}\\ &= S_{yy} - 2 \hat\beta_1 S_{xy} + \hat\beta_1^2 S_{xx} && \text{(definitions of $S_{yy}$, $S_{xy}$, $S_{xx}$)}\\ &= S_{yy} - 2 \frac{S_{xy}^2}{S_{xx}} + \frac{S_{xy}^2}{S_{xx}} && \text{(substitute $\hat\beta_1 = S_{xy} / S_{xx}$)}\\ &= S_{yy} - \frac{S_{xy}^2}{S_{xx}} && \text{(combine like terms)} \end{aligned} \]
The total sum of squares is \(S_{yy}\), so:
\[ \begin{aligned} R^2 &= 1 - \frac{S_{yy} - S_{xy}^2 / S_{xx}}{S_{yy}} && \text{(substitute into the definition of $R^2$)}\\ &= \frac{S_{xy}^2}{S_{xx} S_{yy}} && \text{(simplify)}\\ &= r^2 && \text{($r = S_{xy} / \sqrt{S_{xx} S_{yy}}$)} \end{aligned} \]
Since \(-1 \le r \le 1\) (range of the correlation coefficient), \(0 \le r^2 \le 1\).
Example 6 (\(R^2\) for the regression of glucose on BMI in HERS) For the fit in Example 4, \(R^2\) computed from Definition 8, the square of the Pearson correlation, and lm()’s value agree:
BMI accounts for only about 7% of the variation in baseline fasting glucose.
3.5 Further reading
Linear Models Overview covers linear regression in depth, including inference for the coefficients and multiple predictors. Vittinghoff et al. (2012) cover linear regression in Chapter 4.