---
title: "Splines Lab (stub)"
format: html
---

Stub of the Splines Lab page: the exercises, without the solutions. Solutions: https://morrison-lab.github.io/lds/chapters/splines-lab.html

#### Lab: the scale of wages

Use the [`Wage` data](#dat-wage).
What are the [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) and [sample median](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-median) of `wage`?

::: {.callout-tip}

#### Hint

[Read the file](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas) into a table,
then apply a summary function to one column.

:::

```{python}
# your code here
```

#### Lab: the columns of the Wage data

Use the [`Wage` data](#dat-wage).
In these data, which [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) hold numbers rather than categories?

::: {.callout-tip}

#### Hint

In R, `glimpse()` from `dplyr` prints the name, type and first values of each one.
In Python, the `info()` method of a pandas data frame prints the name and type of each one.

:::

```{python}
# your code here
```

#### Lab: wage and age

Use the [`Wage` data](#dat-wage).
What is the [sample correlation](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-pearson-r) between `wage` and `age`?

::: {.callout-tip}

#### Hint

[Compute the correlation](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas) of two columns with a pandas method in Python, or with `cor()` in R.

:::

```{python}
# your code here
```

#### Lab: a histogram of wages

Use the [`Wage` data](#dat-wage).
Draw a histogram of `wage`.

::: {.callout-tip}

#### Hint

[Draw a histogram](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-plots) of one column, as in the histogram of the year of each car in the primer.

:::

```{python}
# your code here
```

#### Lab: wage against age

Use the [`Wage` data](#dat-wage).
Draw a scatterplot of `wage` against `age`.

::: {.callout-tip}

#### Hint

[Draw a scatterplot](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-plots) of one column against another, as in the plot of acceleration against horsepower in the primer.

:::

```{python}
# your code here
```

#### Lab: plots of every two columns

Use the [`Wage` data](#dat-wage).
Draw a plot of every two different [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) among `age`, `year`, `education` and `wage`, in one grid.

::: {.callout-tip}

#### Hint

In R, the `ggpairs()` function of the `GGally` package draws all those plots in one grid,
even with `education` as categories.
In Python, the `scatter_matrix()` function of `pandas.plotting` draws them for numbers only,
so turn `education` into its level number first.

:::

```{python}
# your code here
```

#### Lab: wage against age and year together

Use the [`Wage` data](#dat-wage).
Draw a three-dimensional [scatterplot](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-scatterplot) of `wage` against `age` and `year`, with every year of the data.

::: {.callout-tip}

#### Hint

The [plotly example in the programming primer](https://morrison-lab.github.io/lds/chapters/programming-primer.html#fig-ggplot2-more-3d) draws a cloud of dots in 3D,
with `plot_ly()` in R and `go.Scatter3d()` in Python.
Turn the plot until you look along the `year` axis, and then along the `age` axis.

:::

```{python}
# your code here
```

#### Lab: predict a wage with a polynomial

Download the Wage data from [`data/wage.csv`](https://morrison-lab.github.io/lds/chapters/data/wage.csv),
which has the `wage` (in thousands of dollars) and `age` of $3000$ male workers in the Mid-Atlantic United States.
Fit a [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) of `wage` on `age` of [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) $4$,
and use it to [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the wage of a $50$-year-old.

::: {.callout-tip}

#### Hint

The [`np.polyfit` and `np.polyval` functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-linalg) [fit](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) and evaluate a [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) of a chosen [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree).
In R, `poly()` supplies the polynomial columns that `lm()` uses.

:::

```{python}
# your code here
```

#### Lab: print the coefficients of a degree-4 polynomial

Fit a [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) of `wage` on `age` of [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) $4$ in the Wage data,
and print a table of its [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter),
with their [standard errors](https://morrison-lab.github.io/sds/estimation.html#def-SE) and [$p$-values](https://morrison-lab.github.io/sds/inference.html#def-p-value).

::: {.callout-tip}

#### Hint

In Python, pass the [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels) `wage ~ age + I(age**2) + I(age**3) + I(age**4)` to `smf.ols()`,
and the fit's `summary2()` holds the table.
In R, the formula `wage ~ poly(age, 4, raw = TRUE)` gives the same fit,
and `summary()` or `tidy()` gives the table.
In a tidymodels recipe, `step_poly()` builds the same terms when its `options` list holds `raw = TRUE`.

:::

```{python}
# your code here
```

#### Lab: build a grid of ages

Build a grid of $100$ evenly spaced ages,
from the youngest worker's age in the Wage data to the oldest's,
and store it in a data frame under the name `age`.

```{python}
# your code here
```

#### Lab: write a helper that plots a fit with confidence bands

Write a helper that fits a [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) of `wage` in the Wage data,
[predicts](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) on the grid of Exercise 10,
and plots the data with the [fitted](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) curve with $95\%$ [confidence](https://morrison-lab.github.io/sds/inference.html#def-confidence-interval) bands.

::: {.callout-tip}

#### Hint

In Python, a function that takes a [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels) can fit it with `smf.ols()`,
and the fit's `get_prediction()` gives the bands.
In R, `predict()` gives the bands:
with `interval = "confidence"` for an `lm()` fit,
and with `type = "conf_int"` for a tidymodels workflow after `fit()`.

:::

```{python}
# your code here
```

#### Lab: plot the degree-4 polynomial fit

Use the helper of Exercise 11 to plot the [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) fit of Exercise 9,
with its $95\%$ [confidence](https://morrison-lab.github.io/sds/inference.html#def-confidence-interval) bands.

```{python}
# your code here
```

#### Lab: choose a polynomial degree by ANOVA

With [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) we must decide on the [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) of the polynomial to use.
The Wage data in [`data/wage.csv`](https://morrison-lab.github.io/lds/chapters/data/wage.csv) record the `wage` (in thousands of dollars) and `age` of $3000$ male workers in the Mid-Atlantic United States.

Fit polynomial [models](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) of `wage` in `age` of degrees 1 through 5,
and use [partial $F$-tests](https://morrison-lab.github.io/lds/chapters/linear-regression-inference.html#def-partial-f-test) of each model against the one before it to decide which degree is enough.

::: {.callout-tip}

#### Hint

The five [models](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) are [nested](https://morrison-lab.github.io/sds/correlation-regression.html#def-nested-regression-models).
The [tidymodels example in the programming primer](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels) compares such models:
`anova_lm()` from `statsmodels.stats.anova` takes the `smf.ols()` fits in order in Python,
and `anova()` takes the `lm()` fits in order in R.

:::

```{python}
# your code here
```

#### Lab: choose the degree with `education` in the models

In Exercise 13, each [model](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) of `wage` in the Wage data held a [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) in `age` alone.
Workers' `education` also varies, as a [categorical variable](https://morrison-lab.github.io/sds/data.html#def-categorical-var) with five levels.

Repeat the [partial $F$-tests](https://morrison-lab.github.io/lds/chapters/linear-regression-inference.html#def-partial-f-test) of Exercise 13,
but with `education` included in each model alongside a polynomial in `age` of [degrees](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) 1, 2 and 3.

::: {.callout-tip}

#### Hint

In a [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels), `education` becomes one [indicator](https://morrison-lab.github.io/mds/indicator-functions.html#def-indicator-function) per category after the first,
in `smf.ols()` in Python and in `lm()` in R.
Compare the fits with `anova_lm()` in Python and `anova()` in R, as in the [tidymodels example in the programming primer](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels).

:::

```{python}
# your code here
```

#### Lab: predict a high wage with a logistic polynomial

Now [model](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) whether a worker in the Wage data earns more than \$250,000 per year,
that is, whether `wage > 250`,
since `wage` is in thousands of dollars.

Fit a [logistic regression](https://morrison-lab.github.io/lds/chapters/logistic-regression.html#def-logistic-regression) of `wage > 250` on a [polynomial](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial) of [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) $4$ in `age`,
and plot the [predicted probability](https://morrison-lab.github.io/sds/classification.html#def-predicted-probability) against age,
with $95\%$ [confidence](https://morrison-lab.github.io/sds/inference.html#def-confidence-interval) bands and a rug plot of the workers.

::: {.callout-tip}

#### Hint

The [GLM example in the programming primer](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels-glm) fits a [logistic regression](https://morrison-lab.github.io/lds/chapters/logistic-regression.html#def-logistic-regression):
`smf.glm()` with a `Binomial` family in Python, and `logistic_reg()` or `glm()` in R.
`get_prediction()` in Python and `predict()` with `se.fit = TRUE` in R give the [standard errors](https://morrison-lab.github.io/sds/estimation.html#def-SE) that the bands need,
and in a tidymodels workflow, `predict()` with `type = "conf_int"` gives the bands themselves.
A rug plot marks each worker's age with a short tick,
at the top of the plot for the workers above \$250,000 and at the bottom for the rest.

:::

```{python}
# your code here
```

#### Lab: predict a wage with a step function

Split the observed ages into $4$ age ranges that each contain the same number of years.
Fit a [regression](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-regression) of `wage` on `age` that gives every worker in an age range the same [fitted](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) value,
and use it to [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the wage of a $50$-year-old.

::: {.callout-tip}

#### Hint

The [`pd.cut` function](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-diagnostics) assigns each value to an interval, with edges from `np.quantile`,
and a [group mean](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas) gives the value of each range.
In R, `cut()` assigns each value to an interval,
with edges from `quantile()`;
in a tidymodels recipe, `step_cut()` does the same.

:::

```{python}
# your code here
```

#### Lab: fit a step function with quantile bins

Split `age` into $4$ [bins](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-bin)
at its [sample quantiles](https://morrison-lab.github.io/sds/nonparametric-models.html#def-sample-quantile) $0.25$, $0.5$ and $0.75$.
Each bin then holds about the same number of workers.
Fit a [step function regression](https://morrison-lab.github.io/lds/chapters/splines.html#def-step-function-regression) of `wage` on these bins,
with one [indicator](https://morrison-lab.github.io/mds/indicator-functions.html#def-indicator-function) for each bin and no [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept).
What does each [coefficient](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) [estimate](https://morrison-lab.github.io/sds/estimation.html#def-estimate)?

::: {.callout-tip}

#### Hint

In Python, `pd.qcut()` cuts a variable at its quantiles,
and `pd.get_dummies()` makes one 0-1 variable for each [bin](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-bin).
In R, `quantile()` gives the cutpoints.
`cut()` assigns each age to its bin,
or `step_discretize()` does so in a tidymodels recipe,
and a `- 1` or `0 +` after the `~` drops the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept).

:::

```{python}
# your code here
```

#### Lab: build a cubic B-spline basis

Build a B-spline [basis](https://morrison-lab.github.io/mds/linear-algebra-subspaces.html#def-basis) of [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) $3$ for `age`
with interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) at the ages $25$, $40$ and $60$,
and include the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept) in the basis.
What size is its [design matrix](https://morrison-lab.github.io/lds/chapters/linear-regression.html#def-design-matrix)?

::: {.callout-tip}

#### Hint

In Python, `sklearn.preprocessing.SplineTransformer()` takes the full list of [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline),
including the boundary ones at the youngest and oldest ages.
In R, `splines::bs()` takes only the interior ones,
and so does `step_bs()` in a tidymodels recipe, through its `options` list.
Setting `intercept = TRUE` keeps the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept).

:::

```{python}
# your code here
```

#### Lab: predict a wage with a regression spline

Fit a [cubic regression spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) of `wage` on `age` with interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) at ages $25$, $40$ and $60$,
and use it to [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the wage of a $50$-year-old.

::: {.callout-tip}

#### Hint

The [`bs` function](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-mgcv) builds a spline basis inside a formula,
and takes the [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) as a setting.
In R, `splines::bs()` builds the spline [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column), with other default settings,
and `step_bs()` builds them in a tidymodels recipe.

:::

```{python}
# your code here
```

#### Lab: let the spline basis place its knots

Exercise 19 fixed the interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) of a [cubic regression spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) of `wage` on `age` at ages $25$, $40$ and $60$.
Instead of specifying knots, a spline [basis](https://morrison-lab.github.io/mds/linear-algebra-subspaces.html#def-basis) can be asked for a number of [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column), `df`,
and left to place the knots itself.

Build a basis for [cubic spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline)s of `age` in the Wage data with `df = 6`,
and find where its interior knots are.

::: {.callout-tip}

#### Hint

In R, `splines::bs()` takes `df`,
as `step_bs()` takes `deg_free` in a tidymodels recipe,
and `attr(basis, "knots")` gives the interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline).
In Python, `SplineTransformer` from scikit-learn takes a number of knots, boundary knots included,
and `knots="quantile"`,
and after `fit()`, its `bsplines_[0].t` lists every knot.
A [basis](https://morrison-lab.github.io/mds/linear-algebra-subspaces.html#def-basis) for [cubic spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline)s with $6$ [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) besides the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept) has $6 - 3 = 3$ interior knots.

:::

```{python}
# your code here
```

#### Lab: fit a degree-0 spline

B-splines need not be [cubic](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-cubic-polynomial).
A spline of [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) $0$ takes one value between each pair of [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline),
like the [step function](https://morrison-lab.github.io/lds/chapters/splines.html#def-step-function-regression) of Exercise 16.

Fit a degree-0 spline of `wage` on `age` in the Wage data with `df = 3`,
and compare its [coefficients](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-parameter) with the [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) `wage` in each quarter of the ages,
split at the [sample quantiles](https://morrison-lab.github.io/sds/nonparametric-models.html#def-sample-quantile) $0.25$, $0.5$ and $0.75$ of `age`.

::: {.callout-tip}

#### Hint

In Python, patsy's `bs(age, df=3, degree=0)` builds the [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) inside an `smf.ols()` [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels),
and `pd.qcut(age, 4)` splits the ages into quarters.
R's `splines::bs()` needs a [degree](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-polynomial-degree) of $1$ or more,
so build the same columns with `cut()` at the values from `quantile()`.

:::

```{python}
# your code here
```

#### Lab: fit a natural spline

A [natural spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-natural-spline) is a [cubic spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) that is linear beyond its boundary [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline),
which steadies the fit at the youngest and oldest ages,
where the Wage data have few workers.

Fit a natural spline of `wage` on `age` with `df = 5`,
and plot the fit over the data, with $95\%$ [confidence](https://morrison-lab.github.io/sds/inference.html#def-confidence-interval) bands.

::: {.callout-tip}

#### Hint

In R, `splines::ns(age, df = 5)` builds the [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) inside the [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels),
and `predict()` with `type = "conf_int"` gives the bands.
In Python, patsy's `cr()` builds a [natural cubic spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-natural-spline) inside an `smf.ols()` formula;
give it the interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) that `ns()` would choose, at the quintiles of `age`,
and `get_prediction()` gives the bands.

:::

```{python}
# your code here
```

#### Lab: predict a wage with a smoothing spline

Fit `wage` on `age` with a [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline)
whose [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) are $4$,
and use it to [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the wage of a $50$-year-old.

::: {.callout-tip}

#### Hint

In R, `smooth.spline()` takes the [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) as `df`
and finds the [smoothing parameter](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) itself.
In Python, `scipy.interpolate.make_smoothing_spline()` takes the smoothing parameter instead,
and needs each distinct age only once,
so pass it the [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) wage at each age,
and the count of workers at each age as `w`.
The effective degrees of freedom are the [trace](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-trace) of the [smoother matrix](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline),
and the smoother matrix times the $j$th [unit vector](https://morrison-lab.github.io/mds/linear-algebra-vectors.html#def-unit-vector) is the fit to that unit vector.

:::

```{python}
# your code here
```

#### Lab: fit a smoothing spline with a fixed smoothing parameter

Fit `wage` on `age` with a [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline)
whose [smoothing parameter](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) $\regpar$ is $0.6$.
How many [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) does the fit have?

::: {.callout-tip}

#### Hint

In Python, reuse `make_smoothing_spline()` and `smoother_df()` from Exercise 23.
In R, `smooth.spline()` rescales `age` to run from $0$ to $1$ before it applies the penalty,
so divide $\regpar$ by the cube of the oldest age minus the youngest to put it on the Python scale.

:::

```{python}
# your code here
```

#### Lab: vary the smoothing parameter

A [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline)'s [smoothing parameter](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) $\regpar$ trades closeness to the data against roughness.
Fit a smoothing spline of `wage` on `age` in the Wage data for each $\regpar$ in $10^{-2}$, $10^{0}$, $10^{2}$, $10^{4}$ and $10^{6}$,
and plot the fits over the data.

::: {.callout-tip}

#### Hint

In Python, reuse `make_smoothing_spline()` and the [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) wage at each age from Exercise 23,
and pass each $\regpar$ as `lam`.
In R, `smooth.spline()` takes $\regpar$ as `lambda`,
but it rescales `age` to run from $0$ to $1$ first,
so pass $\regpar / r^3$,
where $r$ is the largest age minus the smallest, to get the same fit.

:::

```{python}
# your code here
```

#### Lab: choose the smoothing parameter by generalized cross-validation

Instead of fixing the [smoothing parameter](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) $\regpar$,
choose it by minimizing the [generalized cross-validation](https://morrison-lab.github.io/lds/chapters/splines.html#def-gcv) score of the fit of `wage` on `age`,
and add that fit to the plot of Exercise 25.

::: {.callout-tip}

#### Hint

In R, `smooth.spline()` minimizes [generalized cross-validation](https://morrison-lab.github.io/lds/chapters/splines.html#def-gcv)
when it is given neither `df` nor `lambda`.
In Python, `make_smoothing_spline()` with no `lam` scores its own data,
which here are the [sample means](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) at each age rather than the workers,
so compute the score over all $n$ workers,
$n \cdot \RSS / \devof{n}{\trf{S_\regpar}}^2$,
with `smoother_df()` from Exercise 23 for the [trace](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-trace),
and minimize it over $\log_{10} \regpar$.

:::

```{python}
# your code here
```

#### Lab: set how flexible a smoothing spline is

Plot, over the data, the [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) fits of `wage` on `age`
with $1$, $3$, $4$, $8$ and $15$ [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline)
besides the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept).

::: {.callout-tip}

#### Hint

A [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline)'s [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) count the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept),
so solve for the [smoothing parameter](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) that gives one more than each number,
with `brentq()` and `smoother_df()` as in Exercise 23,
or with `df` in `smooth.spline()`.
One besides the intercept needs an infinite smoothing parameter.

:::

```{python}
# your code here
```

#### Lab: compare two spans for local regression

A [LOESS](https://morrison-lab.github.io/lds/chapters/non-parametric-models.html#def-loess) curve fits a line near each age,
using only the share of workers nearest that age,
so its [span](https://morrison-lab.github.io/lds/chapters/non-parametric-models.html#def-span) decides how smooth the curve is,
as the [smoothing parameter](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) does for a [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline).

Plot LOESS curves of `wage` on `age` with spans of $0.2$ and $0.5$ over the data.
How does the smaller span change the curve?

::: {.callout-tip}

#### Hint

In Python, `statsmodels.nonparametric.lowess()` takes the [span](https://morrison-lab.github.io/lds/chapters/non-parametric-models.html#def-span) as `frac`,
the ages at which to evaluate the curve as `xvals`,
and `it=0` to skip its robustness passes.
In R, `loess()` takes the span as `span`;
pass `degree = 1` for a local linear fit and `surface = "direct"` to fit at each age rather than interpolate.

:::

```{python}
# your code here
```

#### Lab: predict a wage with an additive model

Fit an [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of `wage` with three terms:
a [cubic regression spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) of `year` whose interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) are at the median,
a cubic regression spline of `age` with interior knots at the terciles,
and `education` as a [categorical](https://morrison-lab.github.io/sds/data.html#def-categorical-var) variable.
Use it to [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the wage of a $50$-year-old who graduated from college, in $2006$.

::: {.callout-tip}

#### Hint

The [`bs` function](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-mgcv) builds a spline basis for each of the first two terms inside a [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels),
and a column of text is treated as a category automatically.
In R, `splines::bs()` builds the spline columns, with other default settings.

:::

```{python}
# your code here
```

#### Lab: build an additive model by hand

Exercise 29 fit an [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of `wage` through a [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels).
Build one by hand instead:
stack a [natural spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-natural-spline) [basis](https://morrison-lab.github.io/mds/linear-algebra-subspaces.html#def-basis) for `age` with `df = 4`,
a natural spline basis for `year` with `df = 5`,
and an [indicator](https://morrison-lab.github.io/mds/indicator-functions.html#def-indicator-function) [column](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) for each level of `education` into one [model matrix](https://morrison-lab.github.io/lds/chapters/linear-regression.html#def-design-matrix),
and fit `wage` on it by [least squares](https://morrison-lab.github.io/lds/chapters/linear-regression.html#def-least-squares-objective) with no separate [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept).
How many columns does it have?

::: {.callout-tip}

#### Hint

In Python, patsy's `cr()` inside `dmatrix()` builds [natural spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-natural-spline) [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column);
give it the interior [knots](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) that R's `ns()` would choose, at the [sample quantiles](https://morrison-lab.github.io/sds/nonparametric-models.html#def-sample-quantile) $0.25$, $0.5$ and $0.75$ of `age` and $0.2$, $0.4$, $0.6$ and $0.8$ of `year`.
`pd.get_dummies()` builds the [indicator](https://morrison-lab.github.io/mds/indicator-functions.html#def-indicator-function) columns,
and `np.hstack()` stacks the pieces.
In R, `splines::ns()`, `model.matrix(~ education - 1)` and `cbind()` do the same.

:::

```{python}
# your code here
```

#### Lab: plot the partial dependence on age

Plot the age term of the [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of Exercise 30, with $95\%$ [confidence](https://morrison-lab.github.io/sds/inference.html#def-confidence-interval) bands:
[predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) on a [matrix](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix) whose [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column) all sit at their [means](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean),
except the age columns, which run over a grid of $100$ ages,
and shift the curve and its bands down by the curve's mean.

::: {.callout-tip}

#### Hint

In Python, `build_design_matrices()` with the age [columns](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix-column)' `design_info` evaluates them on the grid,
and the fit's `get_prediction()` gives the bands.
In R, `predict()` on an `ns()` [basis](https://morrison-lab.github.io/mds/linear-algebra-subspaces.html#def-basis) evaluates it at new ages,
and the [standard error](https://morrison-lab.github.io/sds/estimation.html#def-SE) of each [prediction](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) $\dprod{\vx}{\hat{\vbeta}}$ is $\sqrt{\qform{V}{\vx}}$,
where $V$ is the fit's `vcov()`.

:::

```{python}
# your code here
```

#### Lab: plot the partial dependence on year

Plot the year term of the [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of Exercise 30 in the same way as the age term in Exercise 31,
over a grid of $100$ years.

```{python}
# your code here
```

#### Lab: plot the age curve of an additive model with smoothing splines

Plot the age [component function](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-component-function) against `age`
for an [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of `wage`
that has a [smoothing spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline) in `age`,
another in `year`,
and `education` as a category.

::: {.callout-tip}

#### Hint

In Python, `pygam` takes a numeric [matrix](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix),
so code `education` by its level numbers with `.cat.codes`,
and build `LinearGAM(s(0) + s(1, n_splines=7) + f(2, lam=0))`.
Here `f()` marks a [categorical](https://morrison-lab.github.io/sds/data.html#def-categorical-var) [feature](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature),
and `year` takes only 7 values,
so its spline gets 7 [basis functions](https://morrison-lab.github.io/lds/chapters/linear-regression.html#def-basis-function).
`partial_dependence(term=0, X=..., width=0.95)` returns the age curve and its $95\%$ bands.
In R, `mgcv::gam()` fits `wage ~ s(age, bs = "cr") + s(year, bs = "cr", k = 7) + education`,
and `predict(..., type = "terms", se.fit = TRUE)` returns each term's curve and the numbers to build its bands from.

:::

```{python}
# your code here
```

#### Lab: give each smoother about 4 degrees of freedom

Plot both [component functions](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-component-function) of the [model](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) of Exercise 33
when its age and year smoothers each have about $4$ [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline).

::: {.callout-tip}

#### Hint

In Python, search for the `lam` at which a smoother of one variable alone has $4 + 1$ [effective degrees of freedom](https://morrison-lab.github.io/lds/chapters/splines.html#def-smoothing-spline),
the $1$ being the [intercept](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-intercept) that `LinearGAM` adds,
and put that `lam` on the matching term before refitting.
In R, `s(age, bs = "cr", k = 5, fx = TRUE)` is a [cubic regression spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) with no penalty and exactly $4$ effective degrees of freedom.

:::

```{python}
# your code here
```

#### Lab: plot the education term

Plot the `education` term of the fit of Exercise 34.
The term holds one number per level of `education`,
so plot one dot per level, with its $95\%$ [confidence](https://morrison-lab.github.io/sds/inference.html#def-confidence-interval) bar,
rather than a curve.

```{python}
# your code here
```

#### Lab: test whether `year` needs a curve

The [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of Exercise 29 gives `year` a curve.
A test can check whether the Wage data support that curve,
or whether a straight line in `year`, or no `year` term at all, does as well.

Consider three [models](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) of `wage`.
Each has a [cubic regression spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) of `age` with `df = 5`.
Each also has `education` as a category.
The models differ only in their `year` term:

- the first has no `year` term;
- the second has a linear term in `year`;
- the third has a cubic regression spline of `year` with `df = 4`.

Each model contains the one before it.

Which of the three models do [partial $F$-tests](https://morrison-lab.github.io/lds/chapters/linear-regression-inference.html#def-partial-f-test) of each model against the one before it favor?

::: {.callout-tip}

#### Hint

The three [models](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) are [nested](https://morrison-lab.github.io/sds/correlation-regression.html#def-nested-regression-models).
The [tidymodels example in the programming primer](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels) compares such models.
In R, `anova()` takes the engines that `extract_fit_engine()` returns, in order.
In Python, `anova_lm()` from `statsmodels.stats.anova` takes the `smf.ols()` fits in order.
Both languages write the spline in the [formula](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-statsmodels) as `bs(x, df = ...)`.

:::

```{python}
# your code here
```

#### Lab: test whether `age` needs a curve

Repeat the tests of Exercise 36 for `age`.
Consider three [models](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-model) of `wage`.
Each has a [cubic regression spline](https://morrison-lab.github.io/lds/chapters/splines.html#def-regression-spline) of `year` with `df = 4`,
and `education` as a category.
The models differ only in their `age` term:

- the first has no `age` term;
- the second has a linear term in `age`;
- the third has a cubic regression spline of `age` with `df = 5`.

Which of the three models do [partial $F$-tests](https://morrison-lab.github.io/lds/chapters/linear-regression-inference.html#def-partial-f-test) of each model against the one before it favor?

::: {.callout-tip}

#### Hint

Reuse the data and the `anova_lm()` and `anova()` calls of Exercise 36,
with the roles of `age` and `year` swapped.

:::

```{python}
# your code here
```

#### Lab: summarize the additive model and predict

Print a summary of the [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of Exercise 34.
Then use it to [predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) `wage` for every worker in the [training set](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-training-set).

::: {.callout-tip}

#### Hint

In Python, `gam_full.summary()` prints the summary,
and `gam_full.predict(Xgam)` gives the [fitted values](https://morrison-lab.github.io/sds/prediction.html#def-fitted-value).
In R, `summary()` and `predict()` do the same for the `mgcv` fit.

:::

```{python}
# your code here
```

#### Lab: fit a logistic additive model for high earners

Call a worker a high earner
when their `wage` is over $250$ thousand dollars.
Fit a [logistic regression](https://morrison-lab.github.io/lds/chapters/logistic-regression.html#def-logistic-regression) [additive model](https://morrison-lab.github.io/lds/chapters/additive-models.html#def-additive-model) of high earning
with the age smoother of Exercise 34,
a linear term in `year`,
and `education` as a category.
Plot the `education` term, as in Exercise 35.

::: {.callout-tip}

#### Hint

In Python, `LogisticGAM(age_term + l_gam(1, lam=0) + f_gam(2, lam=0))` builds it,
with `l_gam` imported from `pygam` as `l`.
In R, `mgcv::gam()` takes `family = binomial`.

:::

```{python}
# your code here
```

#### Lab: count high earners at each education level

Call a worker a high earner when their `wage` is over $250$ thousand dollars.
Count the high earners and the other workers at each level of `education`.
How many high earners have less than a high school education?

::: {.callout-tip}

#### Hint

In Python, `pd.crosstab()` counts the workers in each pair of categories.
In R, `count()` and `pivot_wider()` make the same table.

:::

```{python}
# your code here
```
