Midterm 1 note sheet
Linear regression
| Intercept trick | \(\vfeatmap(\vx) = \tp{\sb{1, x_1, \ldots, x_{\inputdim}}}\) |
| Predictor matrix | \(\mx = \tp{\sb{\vx_1, \ldots, \vx_n}} \in \reals^{n \times \inputdim}\) |
| Design matrix | \(\design(\mx) = \tp{\sb{\vfeatmap(\vx_1), \ldots, \vfeatmap(\vx_n)}}\) |
| Design matrix shape | \(\design(\mx) \in \reals^{n \times p^*}\) |
| Coefficient vector shape | \(\vcoef \in \reals^{p^*}\) |
| Predictions | \(\est{\vy} = \design(\mx) \vcoef\) |
| Risk | \(R(\vcoef) = \E{(\dprod{\vcoef}{\vfeatmap(\vx)} - y)^2}\) |
| Sample MSE | \(\sMSE{\vcoef} = \frac{1}{n} \sumin (\dprod{\vcoef}{\vfeatmap(\vx_i)} - y_i)^2\) |
| Sample MSE, matrix form | \(\sMSE{\vcoef} = \frac{1}{n} \norm{\design(\mx) \vcoef - \vy}_2^2\) |
| Normal equations | \(\stprod{\design(\mx)} \hvcoef = \tprod{\design(\mx)}{\vy}\) |
| Unique solution condition | \(\detf{\stprod{\design(\mx)}} \neq 0\) |
| Solution | \(\hvcoef = (\stprod{\design(\mx)})^{-1} \tprod{\design(\mx)}{\vy}\) |
| Fitted values | \(\est{\vy} = \design(\mx) \hvcoef\) |
| Residual vector | \(\vresid = \vy - \est{\vy}\) |
Feature expansion
| Feature map | \(\vfeatmap : \reals^{\inputdim} \to \reals^{p^*}\) |
| Linear model in the features | \(\modmemf{\vx} = \dprod{\vcoef}{\vfeatmap(\vx)}\) |
| Monomial | \(m(\vx) = \prod_{j=1}^{\inputdim} x_j^{a_j}\) |
| Monomial degree bound | \(\sum_{j=1}^{\inputdim} a_j \le d\) |
Shapes and calculus
| Matrix product shape | \((a \times b)(b \times c) \to a \times c\) |
| Transpose shape | \(a \times b \to b \times a\) |
| Dot product | \(\dprod{\vx}{\vy} = \sumin x_i y_i\) |
| Sample mean | \(\smean{y} = \frac{1}{n} \sumin y_i\) |
| Sum rule | \(\dif{}{w} \sumin u_i = \sumin \dif{u_i}{w}\) |
| Chain rule | \(\dif{h(u)}{w} = \dif{h(u)}{u} \, \dif{u}{w}\) |
| Minimum: first derivative is zero | \(\derivf{\hR}{w} = 0\) |
| Minimum: second derivative is positive | \(\dderivf{\hR}{w} > 0\) |
Vector calculus
| Linear form | \(\deriv{\vcoef} \dprod{\vecf{c}}{\vcoef} = \vecf{c}\) |
| Squared norm | \(\deriv{\vcoef} \dprod{\vcoef}{\vcoef} = 2 \vcoef\) |
| Matrix-vector product, constant \(\matr{A}\) | \(\deriv{\vcoef} (\matr{A} \vv) = (\deriv{\vcoef} \vv) \tp{\matr{A}}\) |
| Special case \(\vv = \vcoef\) | \(\deriv{\vcoef} (\matr{A} \vcoef) = \tp{\matr{A}}\) |
| Next rule needs \(\matr{S}\) symmetric | \(\matr{S} = \tp{\matr{S}}\) |
| Quadratic form | \(\deriv{\vcoef} \qform{\matr{S}}{\vcoef} = 2 \matr{S} \vcoef\) |
Generalization
| Outcome | \(y = \truemeanf{x} + \noise\) |
| Noise variance | \(\Var{\noise} = \varpar\) |
| Bias | \(\Bias{\epredfnf{x}} = \E{\epredfnf{x}} - \truemeanf{x}\) |
| Variance | \(\Var{\epredfnf{x}} = \E{\paren{\epredfnf{x} - \E{\epredfnf{x}}}^2}\) |
| Expected squared error | \(\ESE{x} = \E{\paren{y - \epredfnf{x}}^2}\) |
| Expected squared error decomposition | \(\ESE{x} = \paren{\Bias{\epredfnf{x}}}^2 + \Var{\epredfnf{x}} + \varpar\) |
| Cross-validation estimate, equal-size-fold case | \(\hR_{k\text{-fold}} = \frac{1}{k} \sum_{j=1}^{k} \MSEt_j\) |
| \(\euler\) | \(\e{-1}\) | \(\ln 2\) | \(\ln 3\) | \(\ln 4\) | \(\ln 5\) | \(\ln 10\) | |
|---|---|---|---|---|---|---|---|
| Value | \(2.718\) | \(0.368\) | \(0.693\) | \(1.099\) | \(1.386\) | \(1.609\) | \(2.303\) |