---
title: "Support Vector Machine Lab (stub)"
format: html
---

Stub of the Support Vector Machine Lab page: the exercises, without the solutions. Solutions: https://morrison-lab.github.io/lds/chapters/svm-lab.html

#### Lab: simulate two classes that overlap

Simulate $50$ observations of two [features](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature),
each standard [normal](https://morrison-lab.github.io/pds/random-variables.html#def-normal).
[Label](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable) the first $25$ observations $-1$
and the last $25$ observations $+1$.
Add $1$ to both features of the observations labeled $+1$.

::: {.callout-tip}

#### Hint

The [random number functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-random) draw [normal](https://morrison-lab.github.io/pds/random-variables.html#def-normal) values for a given array shape,
and [array tools](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-arrays) build a vector of [labels](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable) and add a number to selected rows.

:::

```{python}
# your code here
```

#### Lab: plot the simulated classes

Plot the two [features](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) of Exercise 1 against each other.
Color each observation by its [label](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable).

::: {.callout-tip}

#### Hint

A [scatterplot](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-plots) takes one color for each observation,
which you can choose from the [label](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable).

:::

```{python}
# your code here
```

#### Lab: the mean of each feature in each class

Find the [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) of each [feature](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) of Exercise 1 within each [label](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable).

::: {.callout-tip}

#### Hint

[Select the rows](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-arrays) of one [label](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable) with a comparison, then average each column of them.
In R, group the table by label with `group_by()`, then [summarize the columns](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidyverse-more).

:::

```{python}
# your code here
```

#### Lab: fit a support vector classifier

Fit a [support vector](https://morrison-lab.github.io/lds/chapters/svms.html#def-support-vector) classifier with a [linear kernel](https://morrison-lab.github.io/lds/chapters/svms.html#def-kernel-function)
and [cost](https://morrison-lab.github.io/lds/chapters/svms.html#def-c-svm-primal) $C = 10$
to the [features](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) and [labels](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable) of Exercise 1.

::: {.callout-tip}

#### Hint

Every [estimator](https://morrison-lab.github.io/sds/estimation.html#def-estimator) in [scikit-learn](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-sklearn) is [fit](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) with `.fit` on a [matrix](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix) of [features](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) and a vector of [labels](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable);
the settings of a [support vector](https://morrison-lab.github.io/lds/chapters/svms.html#def-support-vector) classifier are its [kernel function](https://morrison-lab.github.io/lds/chapters/svms.html#def-kernel-function) and its [cost](https://morrison-lab.github.io/lds/chapters/svms.html#def-c-svm-primal).
In R, the `svm_linear()` specification of [tidymodels](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels) is fit with `fit()`.

:::

```{python}
# your code here
```

#### Lab: count the support vectors

Find the number of [support vectors](https://morrison-lab.github.io/lds/chapters/svms.html#def-support-vector) of the fit in Exercise 4.

::: {.callout-tip}

#### Hint

A [fitted](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) [estimator](https://morrison-lab.github.io/sds/estimation.html#def-estimator) stores what it learned in [attributes whose names end in `_`](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-sklearn);
one of them holds the [support vectors](https://morrison-lab.github.io/lds/chapters/svms.html#def-support-vector).
In R, `extract_fit_engine()` returns the engine object.
Then `kernlab::nSV()` counts its support vectors.

:::

```{python}
# your code here
```

#### Lab: find the training error

Find the [training error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-training-error) of the fit in Exercise 4,
the proportion of the simulated observations it misclassifies.

::: {.callout-tip}

#### Hint

The [`.predict` method](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-sklearn) returns a [predicted](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) [label](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable) for each row,
and the [training error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-training-error) is the fraction that differ from the actual ones.
In R, `predict()` on a [tidymodels](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels) workflow plays the same role as `.predict`.

:::

```{python}
# your code here
```

#### Lab: count the support vectors at a smaller cost

Refit the classifier of Exercise 4 with the [cost](https://morrison-lab.github.io/lds/chapters/svms.html#def-c-svm-primal) $C = 0.1$,
and find the number of its [support vectors](https://morrison-lab.github.io/lds/chapters/svms.html#def-support-vector).

::: {.callout-tip}

#### Hint

Refit as in Exercise 4 with one setting changed,
then read off the [support vectors](https://morrison-lab.github.io/lds/chapters/svms.html#def-support-vector) as in Exercise 5.

:::

```{python}
# your code here
```

#### Lab: choose the cost by cross-validation

Choose the [cost](https://morrison-lab.github.io/lds/chapters/svms.html#def-c-svm-primal) $C$ from $0.001$, $0.01$, $0.1$, $1$, $5$, $10$ and $100$
by five-fold [cross-validation](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-cross-validation) on the data of Exercise 1,
with [accuracy](https://morrison-lab.github.io/lds/chapters/first-model.html#def-accuracy) as the score.

::: {.callout-tip}

#### Hint

A [loop over the candidate values](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-stdlib), with a [cross-validation](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-cross-validation) score of each, picks the best;
the [`.score` method](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-sklearn) of a [classifier](https://morrison-lab.github.io/sds/classification.html#def-classification-rule) returns its [accuracy](https://morrison-lab.github.io/lds/chapters/first-model.html#def-accuracy).
In R, `tune_grid()` with `vfold_cv()` tries the candidate values in [tidymodels](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels).
Then `select_best()` picks one.

:::

```{python}
# your code here
```

#### Lab: simulate a test set

Simulate $20$ test observations the same way as the [training data](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training-data) of Exercise 1,
$10$ in each [class](https://morrison-lab.github.io/sds/classification.html#def-class).

::: {.callout-tip}

#### Hint

Draw the data with the [random number functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-random) exactly as in Exercise 1, with the new sizes.

:::

```{python}
# your code here
```

#### Lab: find the test error of the tuned fit

[Predict](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) the test observations of Exercise 9 with the fit at the [cost](https://morrison-lab.github.io/lds/chapters/svms.html#def-c-svm-primal) chosen in Exercise 8,
and find the [test error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error).

::: {.callout-tip}

#### Hint

The [`.predict` method](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-sklearn) returns a [predicted](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) [label](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable) for each test row,
and the [test error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-test-error) is the fraction that differ from the actual ones.
In R, `predict()` on a [tidymodels](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels) workflow plays the same role as `.predict`.

:::

```{python}
# your code here
```

#### Lab: simulate two classes with a nonlinear boundary

Simulate $200$ observations of two [features](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature),
each standard [normal](https://morrison-lab.github.io/pds/random-variables.html#def-normal).
Add $2$ to both features of the first $100$ observations,
and subtract $2$ from both features of the next $50$.
[Label](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable) the first $150$ observations $1$ and the last $50$ observations $2$.

::: {.callout-tip}

#### Hint

Build the data with the [random number functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-random) and the [array tools](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-arrays) used in Exercise 1;
shifting a block of rows is a [slice](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-indexing) plus a number.

:::

```{python}
# your code here
```

#### Lab: fit a radial kernel and find its training error

Fit a classifier with a radial [kernel function](https://morrison-lab.github.io/lds/chapters/svms.html#def-kernel-function), $\gamma = 1$ and $C = 1$, to the data of Exercise 11,
and find its [training error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-training-error).

::: {.callout-tip}

#### Hint

Every [estimator](https://morrison-lab.github.io/sds/estimation.html#def-estimator) in [scikit-learn](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-sklearn) is [fit](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) with `.fit`;
here the [kernel function](https://morrison-lab.github.io/lds/chapters/svms.html#def-kernel-function) is radial,
and $\gamma$ and the [cost](https://morrison-lab.github.io/lds/chapters/svms.html#def-c-svm-primal) are its settings.
In R, the `svm_rbf()` specification of [tidymodels](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels) is fit with `fit()`.

:::

```{python}
# your code here
```

#### Lab: find the area under the ROC curve

Find the area under the receiver operating characteristic curve of the radial-kernel fit of Exercise 12 on its [training data](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-training-data),
scoring [class](https://morrison-lab.github.io/sds/classification.html#def-class) $2$ against class $1$.

::: {.callout-tip}

#### Hint

The [metrics functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-sklearn) of scikit-learn take the actual [labels](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable) and a score for each row,
which is the decision function of the [fit](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model).

:::

```{python}
# your code here
```

#### Lab: add a third class

Append $50$ observations of two [features](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature), each standard [normal](https://morrison-lab.github.io/pds/random-variables.html#def-normal), to the data of Exercise 11.
Add $2$ to the second [feature](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-feature) of the new observations,
and [label](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable) them $0$.

::: {.callout-tip}

#### Hint

[Stack rows](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas-more) of new observations under the old ones,
and build the new [labels](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable) with the [array tools](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-arrays).

:::

```{python}
# your code here
```

#### Lab: fit three classes and find the training error

Fit a classifier with a radial [kernel function](https://morrison-lab.github.io/lds/chapters/svms.html#def-kernel-function), $\gamma = 1$ and $C = 10$, to the data of Exercise 14,
and find its [training error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-training-error).

::: {.callout-tip}

#### Hint

Every [estimator](https://morrison-lab.github.io/sds/estimation.html#def-estimator) in [scikit-learn](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-sklearn) is [fit](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-fitted-model) with `.fit`,
and the [training error](https://morrison-lab.github.io/lds/chapters/big-ideas.html#def-training-error) is the fraction of [predicted](https://morrison-lab.github.io/lds/chapters/course-overview.html#def-prediction) [labels](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-output-variable) that differ from the actual ones;
the number of [classes](https://morrison-lab.github.io/sds/classification.html#def-class) does not change the call.
In R, the `svm_rbf()` specification of [tidymodels](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-tidymodels) is fit with `fit()`.

:::

```{python}
# your code here
```
