---
title: "Unsupervised Learning Lab (stub)"
format: html
---

Stub of the Unsupervised Learning Lab page: the exercises, without the solutions. Solutions: https://morrison-lab.github.io/lds/chapters/unsupervised-lab.html

#### Lab: the scale of each arrest variable

Use the [`USArrests` data](#dat-usarrests).
What are the [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) and [sample standard deviation](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-sd) of each of the four numeric variables?

::: {.callout-tip}
#### Hint

[Compute a summary statistic of every column](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas) of a table at once.

:::

```{python}
# your code here
```

#### Lab: murder and assault arrest rates

Use the [`USArrests` data](#dat-usarrests).
What is the [sample correlation](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-pearson-r) between `Murder` and `Assault`?

::: {.callout-tip}
#### Hint

[Compute the correlation](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas) of two columns with a pandas method in Python, or with `cor()` in R.

:::

```{python}
# your code here
```

#### Lab: a histogram of murder arrest rates

Use the [`USArrests` data](#dat-usarrests).
Draw a histogram of `Murder`.

::: {.callout-tip}
#### Hint

[Draw a histogram](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-plots) of one column, as in the histogram of the year of each car in the primer.

:::

```{python}
# your code here
```

#### Lab: murder against assault arrest rates

Use the [`USArrests` data](#dat-usarrests).
Draw a scatterplot of `Murder` against `Assault`.

::: {.callout-tip}
#### Hint

[Draw a scatterplot](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-plots) of one column against another, as in the plot of acceleration against horsepower in the primer.

:::

```{python}
# your code here
```

#### Lab: standardize the arrests data

Standardize the four variables of the USArrests data:
the murder, assault and rape arrest rates per $100{,}000$ residents of each of the $50$ US states, and the percentage of each state's [population](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-population) that lives in cities.

::: {.callout-tip}

#### Hint

A [summary of each column](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-matrices) gives its [mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) and [standard deviation](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-sd),
and [broadcasting](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-matrices) subtracts and divides in one step to [standardize](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-standardization) every column.

:::

```{python}
# your code here
```

#### Lab: find the first principal component

Find the first principal component of the standardized USArrests data from Exercise 5,
and report its loading on each of the four variables.

The sign of a principal component is arbitrary,
so the signs of the [loadings](https://morrison-lab.github.io/lds/chapters/regularization.html#def-derived-feature) can differ between libraries.

::: {.callout-tip}

#### Hint

The [linear algebra functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-linalg) include the [singular value decomposition](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-svd) and the [eigendecomposition](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-eigendecomposition) of a [symmetric](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-symmetric-matrix) [matrix](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix),
which are two routes to the principal components.

:::

```{python}
# your code here
```

#### Lab: find the proportion of variance explained

Find the proportion of the [variance](https://morrison-lab.github.io/pds/variance-covariance.html#def-variance) of the standardized USArrests data from Exercise 5 that each principal component explains.

::: {.callout-tip}

#### Hint

The [variance](https://morrison-lab.github.io/pds/variance-covariance.html#def-variance) along each principal component comes from the [singular values](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-singular-values) or [eigenvalues](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-eigenvalue) of the [linear algebra functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-linalg), as a share of their total.

:::

```{python}
# your code here
```

#### Lab: cluster the states with k-means

Cluster the standardized USArrests data from Exercise 5 into $3$ groups with [k-means](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-kmeans),
starting the [algorithm](https://morrison-lab.github.io/mcs/expressions-algorithms.html#def-algorithm) from $20$ random sets of centers and keeping the best result, <!-- check-term-links: allow -->
and report the size of each cluster.

::: {.callout-tip}

#### Hint

The [`KMeans` estimator](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-sklearn) has a setting for the number of random starts,
and its `labels_` holds the [cluster](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering) of each row.
In R, `kmeans()` has an `nstart` setting for the number of random starts,
and its `cluster` component holds the number of the group of each row.

:::

```{python}
# your code here
```

#### Lab: cluster the states hierarchically

Cluster the standardized USArrests data from Exercise 5 with complete linkage and [Euclidean distance](https://morrison-lab.github.io/mds/linear-algebra-vectors.html#def-euclidean-distance),
cut the dendrogram into $3$ [clusters](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering),
and report the size of each cluster.

::: {.callout-tip}

#### Hint

The [`linkage` and `fcluster` functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-scipy) build the merge tree and cut it into a chosen number of [clusters](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering).
In R, `hclust()` builds the merge tree.
Then `cutree()` cuts it.

:::

```{python}
# your code here
```

#### Lab: find the principal components of gene expression

The `NCI60` data hold how active each of $6830$ genes is in each of $64$ cancer [samples](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample).
Standardize each gene to have [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) $0$ and [standard deviation](https://morrison-lab.github.io/pds/variance-covariance.html#def-sd) $1$,
then compute the principal components of the samples.
Report the proportion of the [variance](https://morrison-lab.github.io/pds/variance-covariance.html#def-variance) explained by each of the first $5$ components.

::: {.callout-tip}

#### Hint

Compute the principal components as in Exercise 6, with the [linear algebra functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-linalg) on the standardized [matrix](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix) of [samples](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample) by genes.

:::

```{python}
# your code here
```

#### Lab: cluster the samples hierarchically

[Cluster](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering) the standardized [samples](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample) from Exercise 10 with complete linkage and [Euclidean distance](https://morrison-lab.github.io/mds/linear-algebra-vectors.html#def-euclidean-distance),
cut the dendrogram into $4$ [clusters](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering),
and report the size of each cluster.

::: {.callout-tip}

#### Hint

The [`linkage` and `fcluster` functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-scipy) build the merge tree and cut it into a chosen number of [clusters](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering), as in Exercise 9.
In R, `hclust()` builds the merge tree.
Then `cutree()` cuts it.

:::

```{python}
# your code here
```

#### Lab: cluster the samples with k-means

[Cluster](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering) the standardized [samples](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample) from Exercise 10 into $4$ groups with [k-means](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-kmeans),
starting the [algorithm](https://morrison-lab.github.io/mcs/expressions-algorithms.html#def-algorithm) from $20$ random sets of centers and keeping the best result, <!-- check-term-links: allow -->
and report the size of each cluster.

::: {.callout-tip}

#### Hint

The [`KMeans` estimator](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-sklearn) has a setting for the number of random starts,
and its `labels_` holds the [cluster](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering) of each row, as in Exercise 8.
In R, `kmeans()` has an `nstart` setting for the number of random starts,
and its `cluster` component holds the number of the group of each row.

:::

```{python}
# your code here
```

#### Lab: compare the k-means and hierarchical clusters

Compare the sizes of the [k-means](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-kmeans) [clusters](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering) from Exercise 12 with the sizes of the hierarchical clusters from Exercise 11.

::: {.callout-tip}

#### Hint

The [counting functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas-more) tabulate each pair of values of two columns, here the two [cluster](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering) assignments of each [sample](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample).

:::

```{python}
# your code here
```
