# Stub of the Regularization Lab page: the exercises, without the solutions. Solutions: https://morrison-lab.github.io/lds/chapters/regularization-lab.html

# Exercise 1
# #### Predicting baseball salaries with ridge regression
#
# The [`Hitters` data](https://morrison-lab.github.io/lds/chapters/neural-nets-lab.html#dat-hitters) give the 1987 opening-day salary, in thousands of dollars,
# of `{r} nrow(na.omit(read.csv("data/hitters.csv")))` Major League Baseball players,
# with `{r} ncol(read.csv("data/hitters.csv")) - 1` other variables:
# their performance statistics in 1986 and over their careers,
# and their league and division.
# We drop players whose salary is missing.
#
# Split the players at random into a training half and a test half.
# Standardize each predictor using the training half.
# For a range of $\regpar$,
# [fit](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) [ridge regression](https://morrison-lab.github.io/lds/chapters/regularization.html#ridge-regression) on the training half with the [closed-form solution](https://morrison-lab.github.io/mcs/expressions-algorithms.html#def-closed-form-solution) of this chapter,
# and compute the sample mean squared error (sMSE) on the test half.



# Exercise 2
# #### Predicting baseball salaries with ridge regression (part 2)
#
# Use the setup of Exercise 1.
#
# Which $\regpar$ has the lowest [test error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error)?



# Exercise 3
# #### Predicting baseball salaries with ridge regression (part 3)
#
# Use the setup of Exercise 1.
#
# How does the [test error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error) of that $\regpar$ compare with that of $\regpar \approx 0$?



# Exercise 4
# #### Predicting baseball salaries with ridge regression (part 4)
#
# Use the setup of Exercise 1.
#
# How does the [test error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error) of that $\regpar$ compare with that of predicting the [training](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training) [mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) salary for every player?



# Exercise 5
# #### Choose the ridge penalty by cross-validation on baseball salaries
#
# In Exercise 1 we chose $\regpar$ by its error on the test half, so no player was left to estimate the error of the choice.
# Choose it using the [training](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training) half alone instead.
#
# Use the training half and the standardized [predictors](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) of Exercise 1.
# Randomly assign the training players to ten [folds](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-cv-folds),
# and for each $\regpar$ of that exercise's grid compute
# the [10-fold cross-validation estimate](https://morrison-lab.github.io/sds/model-validation.html#def-cv-estimate)
# and its [fold-based standard error](https://morrison-lab.github.io/sds/model-validation.html#def-cv-se).



# Exercise 6
# #### Choose the ridge penalty by cross-validation on baseball salaries (part 2)
#
# Use the setup of Exercise 5.
#
# Find the $\regpar$ with the smallest [estimate](https://morrison-lab.github.io/sds/estimation.html#def-estimate),
# and the largest $\regpar$ whose estimate is within one [fold-based standard error](https://morrison-lab.github.io/sds/model-validation.html#def-cv-se) of the smallest
# (the [one-standard-error rule](https://morrison-lab.github.io/sds/model-validation.html#def-one-se-rule)).



# Exercise 7
# #### Choose the ridge penalty by cross-validation on baseball salaries (part 3)
#
# Use the setup of Exercise 5.
#
# Refit on the whole [training](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training) half at each of the two penalties,
# and compute their [test errors](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error).



# Exercise 8
# #### Choose the ridge penalty by cross-validation on baseball salaries (part 4)
#
# Use the setup of Exercise 5.
#
# Do the two penalties differ much in [cross-validation estimate](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-cv-estimate) and in [test error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error)?



# Exercise 9
# #### The coordinate update for the lasso (part 5)
#
# Use the same notation as [the coordinate update exercise](https://morrison-lab.github.io/lds/chapters/regularization.html#exr-lasso-coordinate-update).
#
# Set $\regpar = 0$.
# What does the update reduce to?



# Exercise 10
# #### Count the zero coefficients of ridge and the lasso on baseball salaries
#
# Use the `Hitters` data and the training half of Exercise 1,
# with the predictors standardized and the salary centered by the training mean.
# For $\regpar = 10^2, 10^{2.5}, \dots, 10^5$,
# [fit](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) [ridge regression](https://morrison-lab.github.io/lds/chapters/regularization.html#ridge-regression) with the [closed-form solution](https://morrison-lab.github.io/mcs/expressions-algorithms.html#def-closed-form-solution) of this chapter,
# and fit the [lasso](https://morrison-lab.github.io/lds/chapters/regularization.html#def-lasso) by the [coordinate descent](https://morrison-lab.github.io/lds/chapters/regularization.html#def-coordinate-descent) of [the lasso coordinate update](https://morrison-lab.github.io/lds/chapters/regularization.html#thm-lasso-coordinate-update).



# Exercise 11
# #### Count the zero coefficients of ridge and the lasso on baseball salaries (part 2)
#
# Use the setup of Exercise 10.
#
# For each $\regpar$, count the [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) that are exactly zero in each fit.



# Exercise 12
# #### Count the zero coefficients of ridge and the lasso on baseball salaries (part 3)
#
# Use the setup of Exercise 10.
#
# Which [predictors](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) still have non-zero lasso [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) at $\regpar = 10^4$?



# Exercise 13
# #### Count the zero coefficients of ridge and the lasso on baseball salaries (part 4)
#
# Use the setup of Exercise 10.
#
# How do the counts of exactly zero [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) compare with your [prediction](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) in [the prediction exercise](https://morrison-lab.github.io/lds/chapters/regularization.html#exr-ridge-lasso-zeros-predict)?



# Exercise 14
# #### Choose the lasso penalty by cross-validation on baseball salaries
#
# Use the `Hitters` data and the standardized [training](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training) half of Exercise 10.
# Randomly assign the training players to ten [folds](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-cv-folds).
# For $\regpar = 10^{1.5}, 10^2, \dots, 10^{4.5}$,
# compute the 10-fold [cross-validation estimate](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-cv-estimate) of the [test error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error) of the lasso and of ridge [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression),
# with the lasso computed by the [coordinate descent](https://morrison-lab.github.io/lds/chapters/regularization.html#def-coordinate-descent) of [the lasso coordinate update](https://morrison-lab.github.io/lds/chapters/regularization.html#thm-lasso-coordinate-update).



# Exercise 15
# #### Choose the lasso penalty by cross-validation on baseball salaries (part 2)
#
# Use the setup of Exercise 14.
#
# For each method, find the $\regpar$ with the smallest [estimate](https://morrison-lab.github.io/sds/estimation.html#def-estimate),
# refit on the whole [training](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training) half,
# and compute the [root mean squared error](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-rmse) on the test half.



# Exercise 16
# #### Choose the lasso penalty by cross-validation on baseball salaries (part 3)
#
# Use the setup of Exercise 14.
#
# How many lasso [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) are exactly zero at the chosen $\regpar$?



# Exercise 17
# #### Choose the lasso penalty by cross-validation on baseball salaries (part 4)
#
# Use the setup of Exercise 14.
#
# Which method [predicts](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the test half better?



# Exercise 18
# #### Predict baseball salaries from a few combinations of the predictors
#
# Ridge [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) and the lasso keep all `{r} ncol(x_raw)` [predictors](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) of the `Hitters` data and penalize their [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter).
# Here we instead replace the predictors by $M$ [linear combinations](https://morrison-lab.github.io/mds/linear-algebra-vectors.html#def-linear-combination) of them,
# and fit [least squares](https://morrison-lab.github.io/lds/chapters/linear-regression.html#def-least-squares-objective) on those combinations alone.
# We build the combinations in two ways.
#
# -   *Greatest [variance](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-variance) of the predictors.*
#     Take the first $M$ directions, in order, along which the standardized [training](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training) predictors vary most.
#     These are the first $M$ right singular vectors of the standardized training [predictor](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) matrix
#     (see [principal component analysis](https://morrison-lab.github.io/lds/chapters/unsupervised-learning.html#sec-pca)).
#     Each combination [weights](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) the predictors by the entries of one singular vector.
#
# -   *Relation to the salary.*
#     Weight each standardized predictor by its simple regression [coefficient](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) on the centered salary,
#     and add the weighted predictors to make the first combination.
#     Subtract from every predictor the part of it that the first combination explains,
#     by regressing each predictor on the combination and keeping the [residuals](https://morrison-lab.github.io/lds/chapters/linear-regression.html#def-residual).
#     Build the second combination from the residuals in the same way, and repeat.
#
# Use the training half and the standardized predictors of Exercise 1,
# and the ten [folds](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-cv-folds) of Exercise 5.
# For each way and each $M = 0, 1, \dots, `r ncol(x_raw)`$,
# compute the 10-fold cross-validation estimate.
# Take $M = 0$ to predict the training [mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) salary for every player.



# Exercise 19
# #### Predict baseball salaries from a few combinations of the predictors (part 2)
#
# Use the setup of Exercise 18.
#
# For each way, which $M$ has the smallest [estimate](https://morrison-lab.github.io/sds/estimation.html#def-estimate)?



# Exercise 20
# #### Predict baseball salaries from a few combinations of the predictors (part 3)
#
# Use the setup of Exercise 18.
#
# Compute the [test errors](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error) of the two choices.



# Exercise 21
# #### Predict baseball salaries from a few combinations of the predictors (part 4)
#
# Use the setup of Exercise 18.
#
# How do the [test errors](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error) of the two choices compare with that of ridge [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) at the penalty chosen in Exercise 6?


