Midterm 3 note sheet
| Generative classifier prediction | \(\argmax_k \pi_k f_k(\vx)\) |
| Class prior | \(\pi_k\) |
| Class density | \(f_k(\vx)\) |
| Class posterior | \(\Prf{\Outvar = k \given \vx} = \frac{\pi_k f_k(\vx)}{\sumto{l}{K} \pi_l f_l(\vx)}\) |
| Normal density of one feature | \(f(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \expf{-\frac{(x - \mu)^2}{2 \sigma^2}}\) |
| LDA model | \(\vx \given \Outvar = k \sim \ndist{\vm_k, \Sigma}\) |
| Linear discriminant function | \(\delta_k(\vx) = \tp{\vx} \inv{\Sigma} \vm_k - \frac12 \tp{\vm_k} \inv{\Sigma} \vm_k + \ln \pi_k\) |
| QDA model | \(\vx \given \Outvar = k \sim \ndist{\vm_k, \Sigma_k}\) |
| Quadratic discriminant function | \(\delta_k(\vx) = -\frac12 \ln \abs{\Sigma_k} - \frac12 \tp{(\vx - \vm_k)} \inv{\Sigma_k} (\vx - \vm_k) + \ln \pi_k\) |
| Discriminant prediction | \(\argmax_k \delta_k(\vx)\) |
| Two-class LDA posterior | \(\Prf{\Outvar = 1 \given \vx} = \sigmoidf{\coef{0} + \dprod{\vcoef}{\vx}}\) |
| Its slope vector | \(\vcoef = \inv{\Sigma}(\vm_1 - \vm_0)\) |
| Its intercept | \(\coef{0} = \ln \frac{\pi_1}{\pi_0} - \frac12 \tp{(\vm_1 + \vm_0)} \vcoef\) |
| Logistic function | \(\sigmoidf{z} = \frac{1}{1 + \e{-z}}\) |
| Probability to log odds | \(\prob \underset{\xleftarrow[\odds / (1 + \odds)]{}}{\xrightarrow{\prob / (1 - \prob)}} \odds \underset{\xleftarrow[\e{\logodds}]{}}{\xrightarrow{\ln \odds}} \logodds\) |
| Estimated prior | \(\hat\pi_k = n_k / n\) |
| Estimated class mean | \(\hat{\vm}_k = \frac{1}{n_k} \sum_{i : y_i = k} \vx_i\) |
| Estimated shared covariance | \(\hSigma = \frac{1}{n - K} \sumkK \sum_{i : y_i = k} (\vx_i - \hat{\vm}_k) \tp{(\vx_i - \hat{\vm}_k)}\) |
| Mean squared error | \(\frac1n \sumin (y_i - \hat y_i)^2\) |
| \(K\)-fold cross-validation estimate | \(\frac1K \sum_{k=1}^{K} \text{error}_k\) |
| Error of one fit | \(\text{error}_k = \text{error on fold } k \text{ of the fit to the other } K - 1 \text{ folds}\) |
| Training points per fit | \(\frac{K - 1}{K} n\) |
| Leave-one-out cross-validation | \(K = n\) |
| Ridge regression objective | \(\norm{X \vcoef - \vy}^2 + \regpar \norm{\vcoef}^2\) |
| Ridge minimizer | \(\vcoef_{\text{ridge}} = \inv{\tp{X} X + \regpar I} \tp{X} \vy\) |
| Lasso objective | \(\norm{X \vcoef - \vy}^2 + \regpar \norm{\vcoef}_1\) |
| Lasso penalty norm | \(\norm{\vcoef}_1 = \sum_j \abs{\coef{j}}\) |
| Soft-thresholding operator | \(S_\tau(z) = \sign(z) \maxf{\abs{z} - \tau, 0}\) |
| Orthonormal features | \(\tp{X} X = I\) |
| Least squares coefficient there | \(\hcoef{j} = \dprod{X_{\cdot j}}{\vy}\) |
| Ridge coefficient there | \(\frac{\hcoef{j}}{1 + \regpar}\) |
| Lasso coefficient there | \(S_{\regpar / 2}\paren{\hcoef{j}}\) |
| Inverse of a 2-by-2 matrix | \(\inv{\sbmat{a & b \\ c & d}} = \frac{1}{a d - b c} \sbmat{d & -b \\ -c & a}\) |
| Arguments of the logarithm | \(a, b > 0\) |
| Log of one | \(\ln 1 = 0\) |
| Exponential of zero | \(\e{0} = 1\) |
| Log of an exponential | \(\lnf{\e{s}} = s\) |
| Exponential of a log | \(\e{\ln a} = a\) |
| Log of a reciprocal | \(\lnf{1 / a} = -\ln a\) |
| Exponential of a negative | \(\e{-s} = 1 / \e{s}\) |
| Log of a product | \(\lnf{a b} = \ln a + \ln b\) |
| Log of a quotient | \(\lnf{a / b} = \ln a - \ln b\) |
| Log of a power | \(\lnf{a^k} = k \ln a\) |
| Exponential of a sum | \(\e{s + t} = \e{s} \e{t}\) |
| Exponential of a difference | \(\e{s - t} = \e{s} / \e{t}\) |
| Power of an exponential | \(\paren{\e{s}}^k = \e{k s}\) |
| \(\euler\) | \(\e{-1}\) | \(\e{-2}\) | \(\ln 2\) | \(\ln 3\) | \(\ln 4\) | \(\ln 5\) | \(\ln 10\) | |
|---|---|---|---|---|---|---|---|---|
| Value | \(2.718\) | \(0.368\) | \(0.135\) | \(0.693\) | \(1.099\) | \(1.386\) | \(1.609\) | \(2.303\) |