---
title: "Regularization Lab (stub)"
format: html
---

Stub of the Regularization Lab page: the exercises, without the solutions. Solutions: https://morrison-lab.github.io/lds/chapters/regularization-lab.html

#### Predicting baseball salaries with ridge regression

The [`Hitters` data](https://morrison-lab.github.io/lds/chapters/neural-nets-lab.html#dat-hitters) give the 1987 opening-day salary, in thousands of dollars,
of `{r} nrow(na.omit(read.csv("data/hitters.csv")))` Major League Baseball players,
with `{r} ncol(read.csv("data/hitters.csv")) - 1` other variables:
their performance statistics in 1986 and over their careers,
and their league and division.
We drop players whose salary is missing.

Split the players at random into a training half and a test half.
Standardize each predictor using the training half.
For a range of $\regpar$,
[fit](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) [ridge regression](https://morrison-lab.github.io/lds/chapters/regularization.html#ridge-regression) on the training half with the [closed-form solution](https://morrison-lab.github.io/mcs/expressions-algorithms.html#def-closed-form-solution) of this chapter,
and compute the sample mean squared error (sMSE) on the test half.

```{python}
# your code here
```

#### Predicting baseball salaries with ridge regression (part 2)

Use the setup of Exercise 1.

Which $\regpar$ has the lowest [test error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error)?

```{python}
# your code here
```

#### Predicting baseball salaries with ridge regression (part 3)

Use the setup of Exercise 1.

How does the [test error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error) of that $\regpar$ compare with that of $\regpar \approx 0$?

```{python}
# your code here
```

#### Predicting baseball salaries with ridge regression (part 4)

Use the setup of Exercise 1.

How does the [test error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error) of that $\regpar$ compare with that of predicting the [training](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training) [mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) salary for every player?

```{python}
# your code here
```

#### Choose the ridge penalty by cross-validation on baseball salaries

In Exercise 1 we chose $\regpar$ by its error on the test half, so no player was left to estimate the error of the choice.
Choose it using the [training](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training) half alone instead.

Use the training half and the standardized [predictors](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) of Exercise 1.
Randomly assign the training players to ten [folds](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-cv-folds),
and for each $\regpar$ of that exercise's grid compute
the [10-fold cross-validation estimate](https://morrison-lab.github.io/sds/model-validation.html#def-cv-estimate)
and its [fold-based standard error](https://morrison-lab.github.io/sds/model-validation.html#def-cv-se).

```{python}
# your code here
```

#### Choose the ridge penalty by cross-validation on baseball salaries (part 2)

Use the setup of Exercise 5.

Find the $\regpar$ with the smallest [estimate](https://morrison-lab.github.io/sds/estimation.html#def-estimate),
and the largest $\regpar$ whose estimate is within one [fold-based standard error](https://morrison-lab.github.io/sds/model-validation.html#def-cv-se) of the smallest
(the [one-standard-error rule](https://morrison-lab.github.io/sds/model-validation.html#def-one-se-rule)).

```{python}
# your code here
```

#### Choose the ridge penalty by cross-validation on baseball salaries (part 3)

Use the setup of Exercise 5.

Refit on the whole [training](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training) half at each of the two penalties,
and compute their [test errors](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error).

```{python}
# your code here
```

#### Choose the ridge penalty by cross-validation on baseball salaries (part 4)

Use the setup of Exercise 5.

Do the two penalties differ much in [cross-validation estimate](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-cv-estimate) and in [test error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error)?

```{python}
# your code here
```

#### The coordinate update for the lasso (part 5)

Use the same notation as [the coordinate update exercise](https://morrison-lab.github.io/lds/chapters/regularization.html#exr-lasso-coordinate-update).

Set $\regpar = 0$.
What does the update reduce to?

```{python}
# your code here
```

#### Count the zero coefficients of ridge and the lasso on baseball salaries

Use the `Hitters` data and the training half of Exercise 1,
with the predictors standardized and the salary centered by the training mean.
For $\regpar = 10^2, 10^{2.5}, \dots, 10^5$,
[fit](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) [ridge regression](https://morrison-lab.github.io/lds/chapters/regularization.html#ridge-regression) with the [closed-form solution](https://morrison-lab.github.io/mcs/expressions-algorithms.html#def-closed-form-solution) of this chapter,
and fit the [lasso](https://morrison-lab.github.io/lds/chapters/regularization.html#def-lasso) by the [coordinate descent](https://morrison-lab.github.io/lds/chapters/regularization.html#def-coordinate-descent) of [the lasso coordinate update](https://morrison-lab.github.io/lds/chapters/regularization.html#thm-lasso-coordinate-update).

```{python}
# your code here
```

#### Count the zero coefficients of ridge and the lasso on baseball salaries (part 2)

Use the setup of Exercise 10.

For each $\regpar$, count the [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) that are exactly zero in each fit.

```{python}
# your code here
```

#### Count the zero coefficients of ridge and the lasso on baseball salaries (part 3)

Use the setup of Exercise 10.

Which [predictors](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) still have non-zero lasso [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) at $\regpar = 10^4$?

```{python}
# your code here
```

#### Count the zero coefficients of ridge and the lasso on baseball salaries (part 4)

Use the setup of Exercise 10.

How do the counts of exactly zero [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) compare with your [prediction](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) in [the prediction exercise](https://morrison-lab.github.io/lds/chapters/regularization.html#exr-ridge-lasso-zeros-predict)?

```{python}
# your code here
```

#### Choose the lasso penalty by cross-validation on baseball salaries

Use the `Hitters` data and the standardized [training](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training) half of Exercise 10.
Randomly assign the training players to ten [folds](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-cv-folds).
For $\regpar = 10^{1.5}, 10^2, \dots, 10^{4.5}$,
compute the 10-fold [cross-validation estimate](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-cv-estimate) of the [test error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error) of the lasso and of ridge [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression),
with the lasso computed by the [coordinate descent](https://morrison-lab.github.io/lds/chapters/regularization.html#def-coordinate-descent) of [the lasso coordinate update](https://morrison-lab.github.io/lds/chapters/regularization.html#thm-lasso-coordinate-update).

```{python}
# your code here
```

#### Choose the lasso penalty by cross-validation on baseball salaries (part 2)

Use the setup of Exercise 14.

For each method, find the $\regpar$ with the smallest [estimate](https://morrison-lab.github.io/sds/estimation.html#def-estimate),
refit on the whole [training](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training) half,
and compute the [root mean squared error](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-rmse) on the test half.

```{python}
# your code here
```

#### Choose the lasso penalty by cross-validation on baseball salaries (part 3)

Use the setup of Exercise 14.

How many lasso [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) are exactly zero at the chosen $\regpar$?

```{python}
# your code here
```

#### Choose the lasso penalty by cross-validation on baseball salaries (part 4)

Use the setup of Exercise 14.

Which method [predicts](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the test half better?

```{python}
# your code here
```

#### Predict baseball salaries from a few combinations of the predictors

Ridge [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) and the lasso keep all `{r} ncol(x_raw)` [predictors](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) of the `Hitters` data and penalize their [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter).
Here we instead replace the predictors by $M$ [linear combinations](https://morrison-lab.github.io/mds/linear-algebra-vectors.html#def-linear-combination) of them,
and fit [least squares](https://morrison-lab.github.io/lds/chapters/linear-regression.html#def-least-squares-objective) on those combinations alone.
We build the combinations in two ways.

-   *Greatest [variance](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-variance) of the predictors.*
    Take the first $M$ directions, in order, along which the standardized [training](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training) predictors vary most.
    These are the first $M$ right singular vectors of the standardized training [predictor](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) matrix
    (see [principal component analysis](https://morrison-lab.github.io/lds/chapters/unsupervised-learning.html#sec-pca)).
    Each combination [weights](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) the predictors by the entries of one singular vector.

-   *Relation to the salary.*
    Weight each standardized predictor by its simple regression [coefficient](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) on the centered salary,
    and add the weighted predictors to make the first combination.
    Subtract from every predictor the part of it that the first combination explains,
    by regressing each predictor on the combination and keeping the [residuals](https://morrison-lab.github.io/lds/chapters/linear-regression.html#def-residual).
    Build the second combination from the residuals in the same way, and repeat.

Use the training half and the standardized predictors of Exercise 1,
and the ten [folds](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-cv-folds) of Exercise 5.
For each way and each $M = 0, 1, \dots, `r ncol(x_raw)`$,
compute the 10-fold cross-validation estimate.
Take $M = 0$ to predict the training [mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) salary for every player.

```{python}
# your code here
```

#### Predict baseball salaries from a few combinations of the predictors (part 2)

Use the setup of Exercise 18.

For each way, which $M$ has the smallest [estimate](https://morrison-lab.github.io/sds/estimation.html#def-estimate)?

```{python}
# your code here
```

#### Predict baseball salaries from a few combinations of the predictors (part 3)

Use the setup of Exercise 18.

Compute the [test errors](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error) of the two choices.

```{python}
# your code here
```

#### Predict baseball salaries from a few combinations of the predictors (part 4)

Use the setup of Exercise 18.

How do the [test errors](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error) of the two choices compare with that of ridge [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) at the penalty chosen in Exercise 6?

```{python}
# your code here
```
