Multivariate-Outcome Linear Regression

Published

Last modified: 2026-11-19

Show code
# The Python chunks import numpy and pandas; reticulate provisions them.
library(reticulate) # nolint: undesirable_function_linter
py_packages <- c("numpy", "pandas")
py_require(py_packages)

This material was a section of the linear regression page. It extends the normal equations from one response to several.

1 The multivariate model

DefDefinition 1 (Multivariate linear regression)

Definition 1 (Multivariate linear regression) When there are multiple target variables (\(\noutvars > 1\)), each response vector is \(\vy_i \in \reals^\noutvars\). The linear model assumes:

\[\vy = \tp{\mcoef} \vfeatmap(\vx) + \vnoise, \qquad \vfeatmap(\vx) \in \reals^{p^*},\ \mcoef \in \reals^{p^* \times \noutvars}\]

where noise vectors \(\vnoise \sim \mathcal{N}(\vzero, \covmat)\) have zero expected value and covariance matrix \(\covmat \in \reals^{\noutvars \times \noutvars}\).

In matrix notation over all \(n\) instances:

\[\moutvar = \design \mcoef + \mnoise\]

where \(\moutvar \in \reals^{n \times \noutvars}\) stacks target rows, \(\design \in \reals^{n \times p^*}\) is the design matrix, and \(\mnoise \in \reals^{n \times \noutvars}\) stacks noise rows.

The objective minimizes the squared Frobenius norm of the residual matrix:

\[\est{\mcoef} = \argmin_{\mcoef} \norm{\moutvar - \design \mcoef}_F^2\]

Write \(\moutvar_{*,k}\) and \(\mcoef_{*,k}\) for the \(k\)-th columns of \(\moutvar\) and \(\mcoef\), respectively.

ExmExample 1 (Two targets at once)

Example 1 (Two targets at once) In the Auto data [1], let the targets be mpg and weight, so \(\noutvars = 2\), and let the features be an intercept and horsepower, so \(p^* = 2\). Following Definition 1, build \(\moutvar\), \(\design\), the fitted \(\est{\mcoef}\) and the residual matrix \(\mnoise\), and estimate the noise covariance \(\covmat\) from the residuals:

import numpy as np
import pandas as pd

auto_mr = pd.read_csv("data/auto.csv")
Y_mr = auto_mr[["mpg", "weight"]].to_numpy(float)
Phi_mr = np.column_stack([np.ones(len(Y_mr)), auto_mr["horsepower"]])

# One column of B per target; lstsq fits all columns at once.
B_mr = np.linalg.lstsq(Phi_mr, Y_mr, rcond=None)[0]
E_mr = Y_mr - Phi_mr @ B_mr
# The q x q covariance of the noise: average outer product of the residual rows.
Sigma_mr = E_mr.T @ E_mr / len(Y_mr)
print(Y_mr.shape, Phi_mr.shape, B_mr.shape, E_mr.shape)
(392, 2) (392, 2) (2, 2) (392, 2)
print(np.array2string(Sigma_mr, precision=1, suppress_small=True))
[[    23.9  -1053.2]
 [ -1053.2 181763.8]]
# Bootstrap: resample cars with replacement, refit both targets,
# and recompute the off-diagonal covariance entry each time.
rng_mr = np.random.default_rng(571)
boot_cov_mr = np.empty(2000)
for b in range(2000):
    rows = rng_mr.integers(0, len(Y_mr), len(Y_mr))
    coef_b = np.linalg.lstsq(Phi_mr[rows], Y_mr[rows], rcond=None)[0]
    resid_b = Y_mr[rows] - Phi_mr[rows] @ coef_b
    boot_cov_mr[b] = (resid_b.T @ resid_b / len(rows))[0, 1]
# The 2.5th and 97.5th percentiles of the resampled entries.
cov_ci_mr = np.quantile(boot_cov_mr, [0.025, 0.975])
print(cov_ci_mr)
[-1277.54117641  -822.01445198]
auto_mr <- read.csv("data/auto.csv")
Y_mr <- as.matrix(auto_mr[, c("mpg", "weight")])
Phi_mr <- cbind(1, auto_mr$horsepower)

# One column of B per target; lm with a matrix response fits all columns at once.
B_mr <- coef(lm(Y_mr ~ auto_mr$horsepower))
E_mr <- Y_mr - Phi_mr %*% B_mr
# The q x q covariance of the noise: average outer product of the residual rows.
Sigma_mr <- crossprod(E_mr) / nrow(Y_mr)
rbind(Y = dim(Y_mr), Phi = dim(Phi_mr), B = dim(B_mr), E = dim(E_mr))
    [,1] [,2]
Y    392    2
Phi  392    2
B      2    2
E    392    2
round(Sigma_mr, 1)
           mpg   weight
mpg       23.9  -1053.2
weight -1053.2 181763.8
# Bootstrap: resample cars with replacement, refit both targets,
# and recompute the off-diagonal covariance entry each time.
set.seed(571)
boot_cov_mr <- replicate(2000, {
  rows <- sample(nrow(Y_mr), replace = TRUE)
  coef_b <- coef(lm(Y_mr[rows, ] ~ auto_mr$horsepower[rows]))
  resid_b <- Y_mr[rows, ] - Phi_mr[rows, ] %*% coef_b
  crossprod(resid_b)[1, 2] / length(rows)
})
# The 2.5th and 97.5th percentiles of the resampled entries.
cov_ci_mr <- quantile(boot_cov_mr, c(0.025, 0.975))
cov_ci_mr
      2.5%      97.5% 
-1270.0709  -818.2046 

\(\moutvar\) and \(\mnoise\) have one row per car and one column per target, \(\design\) has one column per feature, and \(\est{\mcoef}\) has one column per target, so its shape is \(p^* \times \noutvars\). The diagonal of the covariance matrix holds the residual variance of each target. The off-diagonal entry, -1053.2, is negative: cars whose mpg is above the line tend to be lighter than the line predicts, so the errors of the two targets are correlated. Its \(95\%\) bootstrap percentile interval, from \(2000\) resamples of the cars, runs from -1270.1 to -818.2 (in miles per gallon times pounds), so the interval excludes zero. The interval describes which cars were drawn, not the process that produced them beyond that. This is the structure the matrix \(\covmat\) in the definition allows for.

2 The objective by columns

LemTheorem 1 (The Frobenius objective is a sum over columns)

Theorem 1 (The Frobenius objective is a sum over columns) For every \(\mcoef \in \reals^{p^* \times \noutvars}\), the objective of Definition 1 is the sum of the squared-error objectives of the \(\noutvars\) single-target regressions:

\[\norm{\moutvar - \design \mcoef}_F^2 = \sumto{k}{\noutvars} \norm{\moutvar_{*,k} - \design \mcoef_{*,k}}^2.\]

Proof. The squared Frobenius norm of a matrix is the sum of the squares of its entries. Sum them column by column. Column \(k\) of \(\moutvar - \design \mcoef\) is \(\moutvar_{*,k} - \design \mcoef_{*,k}\), and the sum of squares of its entries is \(\norm{\moutvar_{*,k} - \design \mcoef_{*,k}}^2\). Adding over \(k = 1, \ldots, \noutvars\) gives the right side.

ExmExample 2 (Two targets from horsepower)

Example 2 (Two targets from horsepower) In the Auto data [1], predict the two targets mpg and acceleration from an intercept and horsepower. For the least squares \(\est{\mcoef}\) and for a second matrix \(\mcoef\) that shifts every entry of \(\est{\mcoef}\) by the same amount, compute the squared Frobenius norm of the residual matrix and the sum of the squared norms of its columns:

import numpy as np
import pandas as pd

auto_mv = pd.read_csv("data/auto.csv")
Phi_mv = np.column_stack([np.ones(len(auto_mv)), auto_mv["horsepower"].to_numpy(float)])
Y_mv = auto_mv[["mpg", "acceleration"]].to_numpy(float)
B_hat_mv = np.linalg.lstsq(Phi_mv, Y_mv, rcond=None)[0]
B_other_mv = B_hat_mv + 0.5

def both_sides(B):
    resid = Y_mv - Phi_mv @ B
    return np.linalg.norm(resid, "fro") ** 2, sum(np.sum(resid[:, k] ** 2) for k in range(resid.shape[1]))

sides_hat = both_sides(B_hat_mv)
sides_other = both_sides(B_other_mv)
print(sides_hat, sides_other)
(np.float64(10948.360306521437), np.float64(10948.360306521441)) (np.float64(2480858.3603065214), np.float64(2480858.3603065214))
auto_mv <- read.csv("data/auto.csv")
Phi_mv <- cbind(1, auto_mv$horsepower)
Y_mv <- as.matrix(auto_mv[, c("mpg", "acceleration")])
B_hat_mv <- solve(crossprod(Phi_mv), crossprod(Phi_mv, Y_mv))
B_other_mv <- B_hat_mv + 0.5

both_sides <- function(B) {
  resid <- Y_mv - Phi_mv %*% B
  c(frobenius = norm(resid, "F")^2, columns = sum(apply(resid, 2, function(col) sum(col^2))))
}
rbind(fitted = both_sides(B_hat_mv), other = both_sides(B_other_mv))
        frobenius    columns
fitted   10948.36   10948.36
other  2480858.36 2480858.36

For the fitted matrix, both sides equal 10,948.4. For the shifted matrix, they equal 2,480,858 and 2,480,858. The identity holds for every \(\mcoef\), not only the minimizer. The fitted matrix has the smaller residual, as the minimizer must.

3 The normal equations

LemTheorem 2 (Multivariate normal equations)

Theorem 2 (Multivariate normal equations) The least-squares estimator for multivariate linear regression is given by:

\[\est{\mcoef} = \inv{\stprod{\design}} \tp{\design} \moutvar\]

Proof. By Theorem 1, \[\norm{\moutvar - \design \mcoef}_F^2 = \sumto{k}{\noutvars} \norm{\moutvar_{*,k} - \design \mcoef_{*,k}}^2,\] so each column \(\mcoef_{*,k}\) of \(\mcoef\) can be optimized independently. By the normal equations, the minimizer for each column is

\[\est{\mcoef}_{*,k} = \inv{\stprod{\design}} \tp{\design} \moutvar_{*,k}.\]

Concatenating all \(\noutvars\) columns yields

\[\est{\mcoef} = \inv{\stprod{\design}} \tp{\design} \moutvar.\]

ExmExample 3 (One formula, three separate fits)

Example 3 (One formula, three separate fits) In the Auto data [1], predict mpg, acceleration and weight from an intercept, horsepower and displacement. Compute \(\est{\mcoef} = (\stprod{\design})^{-1} \tp{\design} \moutvar\) of Theorem 2 in one step, and compare each of its columns with a separate least-squares fit of that target:

import numpy as np
import pandas as pd

auto_mn = pd.read_csv("data/auto.csv")
Y_mn = auto_mn[["mpg", "acceleration", "weight"]].to_numpy(float)
Phi_mn = np.column_stack([np.ones(len(Y_mn)), auto_mn["horsepower"], auto_mn["displacement"]])

# All three targets at once.
B_mn = np.linalg.solve(Phi_mn.T @ Phi_mn, Phi_mn.T @ Y_mn)

# One univariate fit per target column.
B_cols_mn = np.column_stack([
    np.linalg.lstsq(Phi_mn, Y_mn[:, k], rcond=None)[0] for k in range(Y_mn.shape[1])
])
print(np.array2string(B_mn, precision=4, suppress_small=True))
[[  37.4695   21.3115 1380.2336]
 [  -0.0583   -0.074     3.1021]
 [  -0.0408    0.0101    6.5494]]
print(np.allclose(B_mn, B_cols_mn))
True
auto_mn <- read.csv("data/auto.csv")
Y_mn <- as.matrix(auto_mn[, c("mpg", "acceleration", "weight")])
Phi_mn <- cbind(1, auto_mn$horsepower, auto_mn$displacement)

# All three targets at once.
B_mn <- solve(crossprod(Phi_mn), crossprod(Phi_mn, Y_mn))

# One univariate fit per target column.
B_cols_mn <- sapply(colnames(Y_mn), function(k) coef(lm(Y_mn[, k] ~ Phi_mn[, 2] + Phi_mn[, 3])))
round(B_mn, 4)
         mpg acceleration    weight
[1,] 37.4695      21.3115 1380.2336
[2,] -0.0583      -0.0740    3.1021
[3,] -0.0408       0.0101    6.5494
isTRUE(all.equal(unname(B_mn), unname(B_cols_mn)))
[1] TRUE

\(\est{\mcoef}\) has one column per target, and each column matches the fit that ignores the other targets. The multivariate problem decouples this way because the squared Frobenius norm is a sum over the columns of the residual matrix.

4 Videos

VideosWatch

Hutchinson teaches the multivariate case (11 min) [2].

Definitions on this page

Back to top

References

[1]
G. James, D. Witten, T. Hastie, and R. Tibshirani, An introduction to statistical learning with applications in R, 2nd ed. Springer, 2021. Available: https://www.statlearning.com/
[2]
B. Hutchinson, “DATA 471/571 (machine learning) and CSCI 481/581 (deep learning) video lectures.” Western Washington University. Accessed: Sep. 28, 2026. [Online]. Available: https://facultyweb.cs.wwu.edu/~hutchib2/video_lectures/data371/