Midterm 3 note sheet

Download the PDF

Table 1: Discriminant analysis.
Generative classifier prediction \(\argmax_k \pi_k f_k(\vx)\)
Class prior \(\pi_k\)
Class density \(f_k(\vx)\)
Class posterior \(\Prf{\Outvar = k \given \vx} = \frac{\pi_k f_k(\vx)}{\sumto{l}{K} \pi_l f_l(\vx)}\)
Normal density of one feature \(f(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \expf{-\frac{(x - \mu)^2}{2 \sigma^2}}\)
LDA model \(\vx \given \Outvar = k \sim \ndist{\vm_k, \Sigma}\)
Linear discriminant function \(\delta_k(\vx) = \tp{\vx} \inv{\Sigma} \vm_k - \frac12 \tp{\vm_k} \inv{\Sigma} \vm_k + \ln \pi_k\)
QDA model \(\vx \given \Outvar = k \sim \ndist{\vm_k, \Sigma_k}\)
Quadratic discriminant function \(\delta_k(\vx) = -\frac12 \ln \abs{\Sigma_k} - \frac12 \tp{(\vx - \vm_k)} \inv{\Sigma_k} (\vx - \vm_k) + \ln \pi_k\)
Discriminant prediction \(\argmax_k \delta_k(\vx)\)
Two-class LDA posterior \(\Prf{\Outvar = 1 \given \vx} = \sigmoidf{\coef{0} + \dprod{\vcoef}{\vx}}\)
Its slope vector \(\vcoef = \inv{\Sigma}(\vm_1 - \vm_0)\)
Its intercept \(\coef{0} = \ln \frac{\pi_1}{\pi_0} - \frac12 \tp{(\vm_1 + \vm_0)} \vcoef\)
Logistic function \(\sigmoidf{z} = \frac{1}{1 + \e{-z}}\)
Probability to log odds \(\prob \underset{\xleftarrow[\odds / (1 + \odds)]{}}{\xrightarrow{\prob / (1 - \prob)}} \odds \underset{\xleftarrow[\e{\logodds}]{}}{\xrightarrow{\ln \odds}} \logodds\)
Estimated prior \(\hat\pi_k = n_k / n\)
Estimated class mean \(\hat{\vm}_k = \frac{1}{n_k} \sum_{i : y_i = k} \vx_i\)
Estimated shared covariance \(\hSigma = \frac{1}{n - K} \sumkK \sum_{i : y_i = k} (\vx_i - \hat{\vm}_k) \tp{(\vx_i - \hat{\vm}_k)}\)
Table 2: Model selection.
Mean squared error \(\frac1n \sumin (y_i - \hat y_i)^2\)
\(K\)-fold cross-validation estimate \(\frac1K \sum_{k=1}^{K} \text{error}_k\)
Error of one fit \(\text{error}_k = \text{error on fold } k \text{ of the fit to the other } K - 1 \text{ folds}\)
Training points per fit \(\frac{K - 1}{K} n\)
Leave-one-out cross-validation \(K = n\)
Table 3: Regularization.
Ridge regression objective \(\norm{X \vcoef - \vy}^2 + \regpar \norm{\vcoef}^2\)
Ridge minimizer \(\vcoef_{\text{ridge}} = \inv{\tp{X} X + \regpar I} \tp{X} \vy\)
Lasso objective \(\norm{X \vcoef - \vy}^2 + \regpar \norm{\vcoef}_1\)
Lasso penalty norm \(\norm{\vcoef}_1 = \sum_j \abs{\coef{j}}\)
Soft-thresholding operator \(S_\tau(z) = \sign(z) \maxf{\abs{z} - \tau, 0}\)
Orthonormal features \(\tp{X} X = I\)
Least squares coefficient there \(\hcoef{j} = \dprod{X_{\cdot j}}{\vy}\)
Ridge coefficient there \(\frac{\hcoef{j}}{1 + \regpar}\)
Lasso coefficient there \(S_{\regpar / 2}\paren{\hcoef{j}}\)
Inverse of a 2-by-2 matrix \(\inv{\sbmat{a & b \\ c & d}} = \frac{1}{a d - b c} \sbmat{d & -b \\ -c & a}\)
Table 4: Logarithms and exponentials.
Arguments of the logarithm \(a, b > 0\)
Log of one \(\ln 1 = 0\)
Exponential of zero \(\e{0} = 1\)
Log of an exponential \(\lnf{\e{s}} = s\)
Exponential of a log \(\e{\ln a} = a\)
Log of a reciprocal \(\lnf{1 / a} = -\ln a\)
Exponential of a negative \(\e{-s} = 1 / \e{s}\)
Log of a product \(\lnf{a b} = \ln a + \ln b\)
Log of a quotient \(\lnf{a / b} = \ln a - \ln b\)
Log of a power \(\lnf{a^k} = k \ln a\)
Exponential of a sum \(\e{s + t} = \e{s} \e{t}\)
Exponential of a difference \(\e{s - t} = \e{s} / \e{t}\)
Power of an exponential \(\paren{\e{s}}^k = \e{k s}\)
Table 5: Decimal values of exponentials and logarithms.
\(\euler\) \(\e{-1}\) \(\e{-2}\) \(\ln 2\) \(\ln 3\) \(\ln 4\) \(\ln 5\) \(\ln 10\)
Value \(2.718\) \(0.368\) \(0.135\) \(0.693\) \(1.099\) \(1.386\) \(1.609\) \(2.303\)
Back to top