# Stub of the Unsupervised Learning Lab page: the exercises, without the solutions. Solutions: https://morrison-lab.github.io/lds/chapters/unsupervised-lab.html

# Exercise 1
# #### Lab: the scale of each arrest variable
#
# Use the [`USArrests` data](#dat-usarrests).
# What are the [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) and [sample standard deviation](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-sd) of each of the four numeric variables?
#
# ::: {.callout-tip}
# #### Hint
#
# [Compute a summary statistic of every column](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas) of a table at once.
#
# :::



# Exercise 2
# #### Lab: murder and assault arrest rates
#
# Use the [`USArrests` data](#dat-usarrests).
# What is the [sample correlation](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-pearson-r) between `Murder` and `Assault`?
#
# ::: {.callout-tip}
# #### Hint
#
# [Compute the correlation](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas) of two columns with a pandas method in Python, or with `cor()` in R.
#
# :::



# Exercise 3
# #### Lab: a histogram of murder arrest rates
#
# Use the [`USArrests` data](#dat-usarrests).
# Draw a histogram of `Murder`.
#
# ::: {.callout-tip}
# #### Hint
#
# [Draw a histogram](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-plots) of one column, as in the histogram of the year of each car in the primer.
#
# :::



# Exercise 4
# #### Lab: murder against assault arrest rates
#
# Use the [`USArrests` data](#dat-usarrests).
# Draw a scatterplot of `Murder` against `Assault`.
#
# ::: {.callout-tip}
# #### Hint
#
# [Draw a scatterplot](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-plots) of one column against another, as in the plot of acceleration against horsepower in the primer.
#
# :::



# Exercise 5
# #### Lab: standardize the arrests data
#
# Standardize the four variables of the USArrests data:
# the murder, assault and rape arrest rates per $100{,}000$ residents of each of the $50$ US states, and the percentage of each state's [population](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-population) that lives in cities.
#
# ::: {.callout-tip}
#
# #### Hint
#
# A [summary of each column](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-matrices) gives its [mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) and [standard deviation](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-sd),
# and [broadcasting](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-matrices) subtracts and divides in one step to [standardize](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-standardization) every column.
#
# :::



# Exercise 6
# #### Lab: find the first principal component
#
# Find the first principal component of the standardized USArrests data from Exercise 5,
# and report its loading on each of the four variables.
#
# The sign of a principal component is arbitrary,
# so the signs of the [loadings](https://morrison-lab.github.io/lds/chapters/regularization.html#def-derived-feature) can differ between libraries.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [linear algebra functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-linalg) include the [singular value decomposition](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-svd) and the [eigendecomposition](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-eigendecomposition) of a [symmetric](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-symmetric-matrix) [matrix](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix),
# which are two routes to the principal components.
#
# :::



# Exercise 7
# #### Lab: find the proportion of variance explained
#
# Find the proportion of the [variance](https://morrison-lab.github.io/pds/variance-covariance.html#def-variance) of the standardized USArrests data from Exercise 5 that each principal component explains.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [variance](https://morrison-lab.github.io/pds/variance-covariance.html#def-variance) along each principal component comes from the [singular values](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-singular-values) or [eigenvalues](https://morrison-lab.github.io/mds/linear-algebra-special-matrices.html#def-eigenvalue) of the [linear algebra functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-linalg), as a share of their total.
#
# :::



# Exercise 8
# #### Lab: cluster the states with k-means
#
# Cluster the standardized USArrests data from Exercise 5 into $3$ groups with [k-means](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-kmeans),
# starting the [algorithm](https://morrison-lab.github.io/mcs/expressions-algorithms.html#def-algorithm) from $20$ random sets of centers and keeping the best result, <!-- check-term-links: allow -->
# and report the size of each cluster.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [`KMeans` estimator](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-sklearn) has a setting for the number of random starts,
# and its `labels_` holds the [cluster](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering) of each row.
# In R, `kmeans()` has an `nstart` setting for the number of random starts,
# and its `cluster` component holds the number of the group of each row.
#
# :::



# Exercise 9
# #### Lab: cluster the states hierarchically
#
# Cluster the standardized USArrests data from Exercise 5 with complete linkage and [Euclidean distance](https://morrison-lab.github.io/mds/linear-algebra-vectors.html#def-euclidean-distance),
# cut the dendrogram into $3$ [clusters](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering),
# and report the size of each cluster.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [`linkage` and `fcluster` functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-scipy) build the merge tree and cut it into a chosen number of [clusters](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering).
# In R, `hclust()` builds the merge tree.
# Then `cutree()` cuts it.
#
# :::



# Exercise 10
# #### Lab: find the principal components of gene expression
#
# The `NCI60` data hold how active each of $6830$ genes is in each of $64$ cancer [samples](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample).
# Standardize each gene to have [sample mean](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample-mean) $0$ and [standard deviation](https://morrison-lab.github.io/pds/variance-covariance.html#def-sd) $1$,
# then compute the principal components of the samples.
# Report the proportion of the [variance](https://morrison-lab.github.io/pds/variance-covariance.html#def-variance) explained by each of the first $5$ components.
#
# ::: {.callout-tip}
#
# #### Hint
#
# Compute the principal components as in Exercise 6, with the [linear algebra functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-linalg) on the standardized [matrix](https://morrison-lab.github.io/mds/linear-algebra-matrices.html#def-matrix) of [samples](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample) by genes.
#
# :::



# Exercise 11
# #### Lab: cluster the samples hierarchically
#
# [Cluster](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering) the standardized [samples](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample) from Exercise 10 with complete linkage and [Euclidean distance](https://morrison-lab.github.io/mds/linear-algebra-vectors.html#def-euclidean-distance),
# cut the dendrogram into $4$ [clusters](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering),
# and report the size of each cluster.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [`linkage` and `fcluster` functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-scipy) build the merge tree and cut it into a chosen number of [clusters](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering), as in Exercise 9.
# In R, `hclust()` builds the merge tree.
# Then `cutree()` cuts it.
#
# :::



# Exercise 12
# #### Lab: cluster the samples with k-means
#
# [Cluster](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering) the standardized [samples](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample) from Exercise 10 into $4$ groups with [k-means](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-kmeans),
# starting the [algorithm](https://morrison-lab.github.io/mcs/expressions-algorithms.html#def-algorithm) from $20$ random sets of centers and keeping the best result, <!-- check-term-links: allow -->
# and report the size of each cluster.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [`KMeans` estimator](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-sklearn) has a setting for the number of random starts,
# and its `labels_` holds the [cluster](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering) of each row, as in Exercise 8.
# In R, `kmeans()` has an `nstart` setting for the number of random starts,
# and its `cluster` component holds the number of the group of each row.
#
# :::



# Exercise 13
# #### Lab: compare the k-means and hierarchical clusters
#
# Compare the sizes of the [k-means](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-kmeans) [clusters](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering) from Exercise 12 with the sizes of the hierarchical clusters from Exercise 11.
#
# ::: {.callout-tip}
#
# #### Hint
#
# The [counting functions](https://morrison-lab.github.io/lds/chapters/programming-primer.html#exm-prog-pandas-more) tabulate each pair of values of two columns, here the two [cluster](https://morrison-lab.github.io/lds/chapters/anatomy.html#def-clustering) assignments of each [sample](https://morrison-lab.github.io/sds/exploratory-descriptive.html#def-sample).
#
# :::


