# Stub of the Splines Lab page: the exercises, without the solutions. Solutions: https://morrison-lab.github.io/lds/chapters/splines-lab.html

# Exercise 1
# #### Lab: the scale of wages
#
# Use the [`Wage` data](#dat-wage).
# What are the [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) and [sample median](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-median) of `wage`?
#
# ::: {.callout-tip}
#
# #### Hint
#
# [Read the file](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas) into a table,
# then apply a summary function to one column.
#
# :::



# Exercise 2
# #### Lab: the columns of the Wage data
#
# Use the [`Wage` data](#dat-wage).
# In these data, which [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) hold numbers rather than categories?
#
# ::: {.callout-tip}
#
# #### Hint
#
# In R, `glimpse()` from `dplyr` prints the name, type and first values of each one.
# In Python, the `info()` method of a pandas data frame prints the name and type of each one.
#
# :::



# Exercise 3
# #### Lab: wage and age
#
# Use the [`Wage` data](#dat-wage).
# What is the [sample correlation](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-pearson-r) between `wage` and `age`?
#
# ::: {.callout-tip}
#
# #### Hint
#
# [Compute the correlation](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas) of two columns with a pandas method in Python, or with `cor()` in R.
#
# :::



# Exercise 4
# #### Lab: a histogram of wages
#
# Use the [`Wage` data](#dat-wage).
# Draw a histogram of `wage`.
#
# ::: {.callout-tip}
#
# #### Hint
#
# [Draw a histogram](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-plots) of one column, as in the histogram of the year of each car in the primer.
#
# :::



# Exercise 5
# #### Lab: wage against age
#
# Use the [`Wage` data](#dat-wage).
# Draw a scatterplot of `wage` against `age`.
#
# ::: {.callout-tip}
#
# #### Hint
#
# [Draw a scatterplot](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-plots) of one column against another, as in the plot of acceleration against horsepower in the primer.
#
# :::



# Exercise 6
# #### Lab: plots of every two columns
#
# Use the [`Wage` data](#dat-wage).
# Draw a plot of every two different [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) among `age`, `year`, `education` and `wage`, in one grid.
#
# ::: {.callout-tip}
#
# #### Hint
#
# In R, the `ggpairs()` function of the `GGally` package draws all those plots in one grid,
# even with `education` as categories.
# In Python, the `scatter_matrix()` function of `pandas.plotting` draws them for numbers only,
# so turn `education` into its level number first.
#
# :::



# Exercise 7
# #### Lab: wage against age and year together
#
# Use the [`Wage` data](#dat-wage).
# Draw a three-dimensional [scatterplot](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-scatterplot) of `wage` against `age` and `year`, with every year of the data.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [plotly example in the programming primer](https://morrison-lab.github.io/lds/chapters/programming-primer.html#fig-ggplot2-more-3d) draws a cloud of dots in 3D,
# with `plot_ly()` in R and `go.Scatter3d()` in Python.
# Turn the plot until you look along the `year` axis, and then along the `age` axis.
#
# :::



# Exercise 8
# #### Lab: predict a wage with a polynomial
#
# Download the Wage data from [`data/wage.csv`](https://morrison-lab.github.io/lds/chapters/data/wage.csv),
# which has the `wage` (in thousands of dollars) and `age` of $3000$ male workers in the Mid-Atlantic United States.
# Fit a [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) of `wage` on `age` of [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) $4$,
# and use it to [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the wage of a $50$-year-old.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [`np.polyfit` and `np.polyval` functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-linalg) [fit](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) and evaluate a [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) of a chosen [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree).
# In R, `poly()` supplies the polynomial columns that `lm()` uses.
#
# :::



# Exercise 9
# #### Lab: print the coefficients of a degree-4 polynomial
#
# Fit a [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) of `wage` on `age` of [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) $4$ in the Wage data,
# and print a table of its [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter),
# with their [standard errors](https://morrison-lab.github.io/sds/estimation.html#def-SE) and [$p$-values](https://morrison-lab.github.io/sds/inference.html#def-p-value).
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, pass the [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels) `wage ~ age + I(age**2) + I(age**3) + I(age**4)` to `smf.ols()`,
# and the fit's `summary2()` holds the table.
# In R, the formula `wage ~ poly(age, 4, raw = TRUE)` gives the same fit,
# and `summary()` or `tidy()` gives the table.
# In a tidymodels recipe, `step_poly()` builds the same terms when its `options` list holds `raw = TRUE`.
#
# :::



# Exercise 10
# #### Lab: build a grid of ages
#
# Build a grid of $100$ evenly spaced ages,
# from the youngest worker's age in the Wage data to the oldest's,
# and store it in a data frame under the name `age`.



# Exercise 11
# #### Lab: write a helper that plots a fit with confidence bands
#
# Write a helper that fits a [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) of `wage` in the Wage data,
# [predicts](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) on the grid of Exercise 10,
# and plots the data with the [fitted](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) curve with $95\%$ [confidence](https://morrison-lab.github.io/sds/inference.html#def-confidence-interval) bands.
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, a function that takes a [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels) can fit it with `smf.ols()`,
# and the fit's `get_prediction()` gives the bands.
# In R, `predict()` gives the bands:
# with `interval = "confidence"` for an `lm()` fit,
# and with `type = "conf_int"` for a tidymodels workflow after `fit()`.
#
# :::



# Exercise 12
# #### Lab: plot the degree-4 polynomial fit
#
# Use the helper of Exercise 11 to plot the [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) fit of Exercise 9,
# with its $95\%$ [confidence](https://morrison-lab.github.io/sds/inference.html#def-confidence-interval) bands.



# Exercise 13
# #### Lab: choose a polynomial degree by ANOVA
#
# With [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) we must decide on the [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) of the polynomial to use.
# The Wage data in [`data/wage.csv`](https://morrison-lab.github.io/lds/chapters/data/wage.csv) record the `wage` (in thousands of dollars) and `age` of $3000$ male workers in the Mid-Atlantic United States.
#
# Fit polynomial [models](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) of `wage` in `age` of degrees 1 through 5,
# and use [partial $F$-tests](https://morrison-lab.github.io/lds/chapters/linear-regression-inference.html#def-partial-f-test) of each model against the one before it to decide which degree is enough.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The five [models](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) are [nested](https://morrison-lab.github.io/sds/correlation-regression.html#def-nested-regression-models).
# The [tidymodels example in the programming primer](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels) compares such models:
# `anova_lm()` from `statsmodels.stats.anova` takes the `smf.ols()` fits in order in Python,
# and `anova()` takes the `lm()` fits in order in R.
#
# :::



# Exercise 14
# #### Lab: choose the degree with `education` in the models
#
# In Exercise 13, each [model](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) of `wage` in the Wage data held a [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) in `age` alone.
# Workers' `education` also varies, as a [categorical variable](https://morrison-lab.github.io/sds/data.html#def-categorical-var) with five levels.
#
# Repeat the [partial $F$-tests](https://morrison-lab.github.io/lds/chapters/linear-regression-inference.html#def-partial-f-test) of Exercise 13,
# but with `education` included in each model alongside a polynomial in `age` of [degrees](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) 1, 2 and 3.
#
# ::: {.callout-tip}
#
# #### Hint
#
# In a [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels), `education` becomes one [indicator](https://morrison-lab.github.io/mds/indicator-functions.html#def-indicator-function) per category after the first,
# in `smf.ols()` in Python and in `lm()` in R.
# Compare the fits with `anova_lm()` in Python and `anova()` in R, as in the [tidymodels example in the programming primer](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels).
#
# :::



# Exercise 15
# #### Lab: predict a high wage with a logistic polynomial
#
# Now [model](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) whether a worker in the Wage data earns more than \$250,000 per year,
# that is, whether `wage > 250`,
# since `wage` is in thousands of dollars.
#
# Fit a [logistic regression](https://morrison-lab.github.io/lds/chapters/logistic-regression.html#def-logistic-regression) of `wage > 250` on a [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) of [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) $4$ in `age`,
# and plot the [predicted probability](https://morrison-lab.github.io/sds/classification.html#def-predicted-probability) against age,
# with $95\%$ [confidence](https://morrison-lab.github.io/sds/inference.html#def-confidence-interval) bands and a rug plot of the workers.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [GLM example in the programming primer](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels-glm) fits a [logistic regression](https://morrison-lab.github.io/lds/chapters/logistic-regression.html#def-logistic-regression):
# `smf.glm()` with a `Binomial` family in Python, and `logistic_reg()` or `glm()` in R.
# `get_prediction()` in Python and `predict()` with `se.fit = TRUE` in R give the [standard errors](https://morrison-lab.github.io/sds/estimation.html#def-SE) that the bands need,
# and in a tidymodels workflow, `predict()` with `type = "conf_int"` gives the bands themselves.
# A rug plot marks each worker's age with a short tick,
# at the top of the plot for the workers above \$250,000 and at the bottom for the rest.
#
# :::



# Exercise 16
# #### Lab: predict a wage with a step function
#
# Split the observed ages into $4$ age ranges that each contain the same number of years.
# Fit a [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) of `wage` on `age` that gives every worker in an age range the same [fitted](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) value,
# and use it to [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the wage of a $50$-year-old.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [`pd.cut` function](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-diagnostics) assigns each value to an interval, with edges from `np.quantile`,
# and a [group mean](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas) gives the value of each range.
# In R, `cut()` assigns each value to an interval,
# with edges from `quantile()`;
# in a tidymodels recipe, `step_cut()` does the same.
#
# :::



# Exercise 17
# #### Lab: fit a step function with quantile bins
#
# Split `age` into $4$ [bins](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-bin)
# at its [sample quantiles](https://morrison-lab.github.io/sds/nonparametric-models.html#def-sample-quantile) $0.25$, $0.5$ and $0.75$.
# Each bin then holds about the same number of workers.
# Fit a [step function regression](https://morrison-lab.github.io/lds/chapters/splines.html#def-step-function-regression) of `wage` on these bins,
# with one [indicator](https://morrison-lab.github.io/mds/indicator-functions.html#def-indicator-function) for each bin and no [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept).
# What does each [coefficient](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) [estimate](https://morrison-lab.github.io/sds/estimation.html#def-estimate)?
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, `pd.qcut()` cuts a variable at its quantiles,
# and `pd.get_dummies()` makes one 0-1 variable for each [bin](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-bin).
# In R, `quantile()` gives the cutpoints.
# `cut()` assigns each age to its bin,
# or `step_discretize()` does so in a tidymodels recipe,
# and a `- 1` or `0 +` after the `~` drops the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept).
#
# :::



# Exercise 18
# #### Lab: build a cubic B-spline basis
#
# Build a B-spline [basis](https://morrison-lab.github.io/mds/linear-algebra-subspaces.html#def-basis) of [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) $3$ for `age`
# with interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) at the ages $25$, $40$ and $60$,
# and include the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept) in the basis.
# What size is its [design matrix](https://morrison-lab.github.io/lds/chapters/linear-regression.html#def-design-matrix)?
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, `sklearn.preprocessing.SplineTransformer()` takes the full list of [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline),
# including the boundary ones at the youngest and oldest ages.
# In R, `splines::bs()` takes only the interior ones,
# and so does `step_bs()` in a tidymodels recipe, through its `options` list.
# Setting `intercept = TRUE` keeps the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept).
#
# :::



# Exercise 19
# #### Lab: predict a wage with a regression spline
#
# Fit a [cubic regression spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) of `wage` on `age` with interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) at ages $25$, $40$ and $60$,
# and use it to [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the wage of a $50$-year-old.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [`bs` function](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-mgcv) builds a spline basis inside a formula,
# and takes the [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) as a setting.
# In R, `splines::bs()` builds the spline [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column), with other default settings,
# and `step_bs()` builds them in a tidymodels recipe.
#
# :::



# Exercise 20
# #### Lab: let the spline basis place its knots
#
# Exercise 19 fixed the interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) of a [cubic regression spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) of `wage` on `age` at ages $25$, $40$ and $60$.
# Instead of specifying knots, a spline [basis](https://morrison-lab.github.io/mds/linear-algebra-subspaces.html#def-basis) can be asked for a number of [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column), `df`,
# and left to place the knots itself.
#
# Build a basis for [cubic spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline)s of `age` in the Wage data with `df = 6`,
# and find where its interior knots are.
#
# ::: {.callout-tip}
#
# #### Hint
#
# In R, `splines::bs()` takes `df`,
# as `step_bs()` takes `deg_free` in a tidymodels recipe,
# and `attr(basis, "knots")` gives the interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline).
# In Python, `SplineTransformer` from scikit-learn takes a number of knots, boundary knots included,
# and `knots="quantile"`,
# and after `fit()`, its `bsplines_[0].t` lists every knot.
# A [basis](https://morrison-lab.github.io/mds/linear-algebra-subspaces.html#def-basis) for [cubic spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline)s with $6$ [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) besides the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept) has $6 - 3 = 3$ interior knots.
#
# :::



# Exercise 21
# #### Lab: fit a degree-0 spline
#
# B-splines need not be [cubic](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-cubic-polynomial).
# A spline of [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) $0$ takes one value between each pair of [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline),
# like the [step function](https://morrison-lab.github.io/lds/chapters/splines.html#def-step-function-regression) of Exercise 16.
#
# Fit a degree-0 spline of `wage` on `age` in the Wage data with `df = 3`,
# and compare its [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) with the [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) `wage` in each quarter of the ages,
# split at the [sample quantiles](https://morrison-lab.github.io/sds/nonparametric-models.html#def-sample-quantile) $0.25$, $0.5$ and $0.75$ of `age`.
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, patsy's `bs(age, df=3, degree=0)` builds the [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) inside an `smf.ols()` [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels),
# and `pd.qcut(age, 4)` splits the ages into quarters.
# R's `splines::bs()` needs a [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) of $1$ or more,
# so build the same columns with `cut()` at the values from `quantile()`.
#
# :::



# Exercise 22
# #### Lab: fit a natural spline
#
# A [natural spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-natural-spline) is a [cubic spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) that is linear beyond its boundary [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline),
# which steadies the fit at the youngest and oldest ages,
# where the Wage data have few workers.
#
# Fit a natural spline of `wage` on `age` with `df = 5`,
# and plot the fit over the data, with $95\%$ [confidence](https://morrison-lab.github.io/sds/inference.html#def-confidence-interval) bands.
#
# ::: {.callout-tip}
#
# #### Hint
#
# In R, `splines::ns(age, df = 5)` builds the [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) inside the [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels),
# and `predict()` with `type = "conf_int"` gives the bands.
# In Python, patsy's `cr()` builds a [natural cubic spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-natural-spline) inside an `smf.ols()` formula;
# give it the interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) that `ns()` would choose, at the quintiles of `age`,
# and `get_prediction()` gives the bands.
#
# :::



# Exercise 23
# #### Lab: predict a wage with a smoothing spline
#
# Fit `wage` on `age` with a [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline)
# whose [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) are $4$,
# and use it to [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the wage of a $50$-year-old.
#
# ::: {.callout-tip}
#
# #### Hint
#
# In R, `smooth.spline()` takes the [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) as `df`
# and finds the [smoothing parameter](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) itself.
# In Python, `scipy.interpolate.make_smoothing_spline()` takes the smoothing parameter instead,
# and needs each distinct age only once,
# so pass it the [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) wage at each age,
# and the count of workers at each age as `w`.
# The effective degrees of freedom are the [trace](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-trace) of the [smoother matrix](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline),
# and the smoother matrix times the $j$th [unit vector](https://morrison-lab.github.io/mds/linear-algebra-vectors.html#def-unit-vector) is the fit to that unit vector.
#
# :::



# Exercise 24
# #### Lab: fit a smoothing spline with a fixed smoothing parameter
#
# Fit `wage` on `age` with a [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline)
# whose [smoothing parameter](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) $\regpar$ is $0.6$.
# How many [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) does the fit have?
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, reuse `make_smoothing_spline()` and `smoother_df()` from Exercise 23.
# In R, `smooth.spline()` rescales `age` to run from $0$ to $1$ before it applies the penalty,
# so divide $\regpar$ by the cube of the oldest age minus the youngest to put it on the Python scale.
#
# :::



# Exercise 25
# #### Lab: vary the smoothing parameter
#
# A [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline)'s [smoothing parameter](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) $\regpar$ trades closeness to the data against roughness.
# Fit a smoothing spline of `wage` on `age` in the Wage data for each $\regpar$ in $10^{-2}$, $10^{0}$, $10^{2}$, $10^{4}$ and $10^{6}$,
# and plot the fits over the data.
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, reuse `make_smoothing_spline()` and the [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) wage at each age from Exercise 23,
# and pass each $\regpar$ as `lam`.
# In R, `smooth.spline()` takes $\regpar$ as `lambda`,
# but it rescales `age` to run from $0$ to $1$ first,
# so pass $\regpar / r^3$,
# where $r$ is the largest age minus the smallest, to get the same fit.
#
# :::



# Exercise 26
# #### Lab: choose the smoothing parameter by generalized cross-validation
#
# Instead of fixing the [smoothing parameter](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) $\regpar$,
# choose it by minimizing the [generalized cross-validation](https://morrison-lab.github.io/lds/chapters/splines.html#def-gcv) score of the fit of `wage` on `age`,
# and add that fit to the plot of Exercise 25.
#
# ::: {.callout-tip}
#
# #### Hint
#
# In R, `smooth.spline()` minimizes [generalized cross-validation](https://morrison-lab.github.io/lds/chapters/splines.html#def-gcv)
# when it is given neither `df` nor `lambda`.
# In Python, `make_smoothing_spline()` with no `lam` scores its own data,
# which here are the [sample means](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) at each age rather than the workers,
# so compute the score over all $n$ workers,
# $n \cdot \RSS / \devof{n}{\trf{S_\regpar}}^2$,
# with `smoother_df()` from Exercise 23 for the [trace](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-trace),
# and minimize it over $\log_{10} \regpar$.
#
# :::



# Exercise 27
# #### Lab: set how flexible a smoothing spline is
#
# Plot, over the data, the [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) fits of `wage` on `age`
# with $1$, $3$, $4$, $8$ and $15$ [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline)
# besides the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept).
#
# ::: {.callout-tip}
#
# #### Hint
#
# A [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline)'s [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) count the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept),
# so solve for the [smoothing parameter](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) that gives one more than each number,
# with `brentq()` and `smoother_df()` as in Exercise 23,
# or with `df` in `smooth.spline()`.
# One besides the intercept needs an infinite smoothing parameter.
#
# :::



# Exercise 28
# #### Lab: compare two spans for local regression
#
# A [LOESS](https://morrison-lab.github.io/lds/chapters/non-parametric-models.html#def-loess) curve fits a line near each age,
# using only the share of workers nearest that age,
# so its [span](https://morrison-lab.github.io/lds/chapters/non-parametric-models.html#def-span) decides how smooth the curve is,
# as the [smoothing parameter](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) does for a [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline).
#
# Plot LOESS curves of `wage` on `age` with spans of $0.2$ and $0.5$ over the data.
# How does the smaller span change the curve?
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, `statsmodels.nonparametric.lowess()` takes the [span](https://morrison-lab.github.io/lds/chapters/non-parametric-models.html#def-span) as `frac`,
# the ages at which to evaluate the curve as `xvals`,
# and `it=0` to skip its robustness passes.
# In R, `loess()` takes the span as `span`;
# pass `degree = 1` for a local linear fit and `surface = "direct"` to fit at each age rather than interpolate.
#
# :::



# Exercise 29
# #### Lab: predict a wage with an additive model
#
# Fit an [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of `wage` with three terms:
# a [cubic regression spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) of `year` whose interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) are at the median,
# a cubic regression spline of `age` with interior knots at the terciles,
# and `education` as a [categorical](https://morrison-lab.github.io/sds/data.html#def-categorical-var) variable.
# Use it to [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the wage of a $50$-year-old who graduated from college, in $2006$.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [`bs` function](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-mgcv) builds a spline basis for each of the first two terms inside a [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels),
# and a column of text is treated as a category automatically.
# In R, `splines::bs()` builds the spline columns, with other default settings.
#
# :::



# Exercise 30
# #### Lab: build an additive model by hand
#
# Exercise 29 fit an [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of `wage` through a [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels).
# Build one by hand instead:
# stack a [natural spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-natural-spline) [basis](https://morrison-lab.github.io/mds/linear-algebra-subspaces.html#def-basis) for `age` with `df = 4`,
# a natural spline basis for `year` with `df = 5`,
# and an [indicator](https://morrison-lab.github.io/mds/indicator-functions.html#def-indicator-function) [column](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) for each level of `education` into one [model matrix](https://morrison-lab.github.io/lds/chapters/linear-regression.html#def-design-matrix),
# and fit `wage` on it by [least squares](https://morrison-lab.github.io/lds/chapters/linear-regression.html#def-least-squares-objective) with no separate [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept).
# How many columns does it have?
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, patsy's `cr()` inside `dmatrix()` builds [natural spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-natural-spline) [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column);
# give it the interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) that R's `ns()` would choose, at the [sample quantiles](https://morrison-lab.github.io/sds/nonparametric-models.html#def-sample-quantile) $0.25$, $0.5$ and $0.75$ of `age` and $0.2$, $0.4$, $0.6$ and $0.8$ of `year`.
# `pd.get_dummies()` builds the [indicator](https://morrison-lab.github.io/mds/indicator-functions.html#def-indicator-function) columns,
# and `np.hstack()` stacks the pieces.
# In R, `splines::ns()`, `model.matrix(~ education - 1)` and `cbind()` do the same.
#
# :::



# Exercise 31
# #### Lab: plot the partial dependence on age
#
# Plot the age term of the [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of Exercise 30, with $95\%$ [confidence](https://morrison-lab.github.io/sds/inference.html#def-confidence-interval) bands:
# [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) on a [matrix](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix) whose [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) all sit at their [means](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean),
# except the age columns, which run over a grid of $100$ ages,
# and shift the curve and its bands down by the curve's mean.
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, `build_design_matrices()` with the age [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column)' `design_info` evaluates them on the grid,
# and the fit's `get_prediction()` gives the bands.
# In R, `predict()` on an `ns()` [basis](https://morrison-lab.github.io/mds/linear-algebra-subspaces.html#def-basis) evaluates it at new ages,
# and the [standard error](https://morrison-lab.github.io/sds/estimation.html#def-SE) of each [prediction](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) $\dprod{\vx}{\hat{\vbeta}}$ is $\sqrt{\qform{V}{\vx}}$,
# where $V$ is the fit's `vcov()`.
#
# :::



# Exercise 32
# #### Lab: plot the partial dependence on year
#
# Plot the year term of the [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of Exercise 30 in the same way as the age term in Exercise 31,
# over a grid of $100$ years.



# Exercise 33
# #### Lab: plot the age curve of an additive model with smoothing splines
#
# Plot the age [component function](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-component-function) against `age`
# for an [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of `wage`
# that has a [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) in `age`,
# another in `year`,
# and `education` as a category.
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, `pygam` takes a numeric [matrix](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix),
# so code `education` by its level numbers with `.cat.codes`,
# and build `LinearGAM(s(0) + s(1, n_splines=7) + f(2, lam=0))`.
# Here `f()` marks a [categorical](https://morrison-lab.github.io/sds/data.html#def-categorical-var) [feature](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature),
# and `year` takes only 7 values,
# so its spline gets 7 [basis functions](https://morrison-lab.github.io/lds/chapters/linear-regression.html#def-basis-function).
# `partial_dependence(term=0, X=..., width=0.95)` returns the age curve and its $95\%$ bands.
# In R, `mgcv::gam()` fits `wage ~ s(age, bs = "cr") + s(year, bs = "cr", k = 7) + education`,
# and `predict(..., type = "terms", se.fit = TRUE)` returns each term's curve and the numbers to build its bands from.
#
# :::



# Exercise 34
# #### Lab: give each smoother about 4 degrees of freedom
#
# Plot both [component functions](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-component-function) of the [model](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) of Exercise 33
# when its age and year smoothers each have about $4$ [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline).
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, search for the `lam` at which a smoother of one variable alone has $4 + 1$ [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline),
# the $1$ being the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept) that `LinearGAM` adds,
# and put that `lam` on the matching term before refitting.
# In R, `s(age, bs = "cr", k = 5, fx = TRUE)` is a [cubic regression spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) with no penalty and exactly $4$ effective degrees of freedom.
#
# :::



# Exercise 35
# #### Lab: plot the education term
#
# Plot the `education` term of the fit of Exercise 34.
# The term holds one number per level of `education`,
# so plot one dot per level, with its $95\%$ [confidence](https://morrison-lab.github.io/sds/inference.html#def-confidence-interval) bar,
# rather than a curve.



# Exercise 36
# #### Lab: test whether `year` needs a curve
#
# The [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of Exercise 29 gives `year` a curve.
# A test can check whether the Wage data support that curve,
# or whether a straight line in `year`, or no `year` term at all, does as well.
#
# Consider three [models](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) of `wage`.
# Each has a [cubic regression spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) of `age` with `df = 5`.
# Each also has `education` as a category.
# The models differ only in their `year` term:
#
# - the first has no `year` term;
# - the second has a linear term in `year`;
# - the third has a cubic regression spline of `year` with `df = 4`.
#
# Each model contains the one before it.
#
# Which of the three models do [partial $F$-tests](https://morrison-lab.github.io/lds/chapters/linear-regression-inference.html#def-partial-f-test) of each model against the one before it favor?
#
# ::: {.callout-tip}
#
# #### Hint
#
# The three [models](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) are [nested](https://morrison-lab.github.io/sds/correlation-regression.html#def-nested-regression-models).
# The [tidymodels example in the programming primer](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels) compares such models.
# In R, `anova()` takes the engines that `extract_fit_engine()` returns, in order.
# In Python, `anova_lm()` from `statsmodels.stats.anova` takes the `smf.ols()` fits in order.
# Both languages write the spline in the [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels) as `bs(x, df = ...)`.
#
# :::



# Exercise 37
# #### Lab: test whether `age` needs a curve
#
# Repeat the tests of Exercise 36 for `age`.
# Consider three [models](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) of `wage`.
# Each has a [cubic regression spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) of `year` with `df = 4`,
# and `education` as a category.
# The models differ only in their `age` term:
#
# - the first has no `age` term;
# - the second has a linear term in `age`;
# - the third has a cubic regression spline of `age` with `df = 5`.
#
# Which of the three models do [partial $F$-tests](https://morrison-lab.github.io/lds/chapters/linear-regression-inference.html#def-partial-f-test) of each model against the one before it favor?
#
# ::: {.callout-tip}
#
# #### Hint
#
# Reuse the data and the `anova_lm()` and `anova()` calls of Exercise 36,
# with the roles of `age` and `year` swapped.
#
# :::



# Exercise 38
# #### Lab: summarize the additive model and predict
#
# Print a summary of the [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of Exercise 34.
# Then use it to [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) `wage` for every worker in the [training set](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-training-set).
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, `gam_full.summary()` prints the summary,
# and `gam_full.predict(Xgam)` gives the [fitted values](https://morrison-lab.github.io/sds/prediction.html#def-fitted-value).
# In R, `summary()` and `predict()` do the same for the `mgcv` fit.
#
# :::



# Exercise 39
# #### Lab: fit a logistic additive model for high earners
#
# Call a worker a high earner
# when their `wage` is over $250$ thousand dollars.
# Fit a [logistic regression](https://morrison-lab.github.io/lds/chapters/logistic-regression.html#def-logistic-regression) [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of high earning
# with the age smoother of Exercise 34,
# a linear term in `year`,
# and `education` as a category.
# Plot the `education` term, as in Exercise 35.
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, `LogisticGAM(age_term + l_gam(1, lam=0) + f_gam(2, lam=0))` builds it,
# with `l_gam` imported from `pygam` as `l`.
# In R, `mgcv::gam()` takes `family = binomial`.
#
# :::



# Exercise 40
# #### Lab: count high earners at each education level
#
# Call a worker a high earner when their `wage` is over $250$ thousand dollars.
# Count the high earners and the other workers at each level of `education`.
# How many high earners have less than a high school education?
#
# ::: {.callout-tip}
#
# #### Hint
#
# In Python, `pd.crosstab()` counts the workers in each pair of categories.
# In R, `count()` and `pivot_wider()` make the same table.
#
# :::


