Midterm 2 note sheet
| Confusion matrix rows: the true class | \(Y\) |
| Confusion matrix columns: the predicted class | \(\hat Y\) |
| True positives | \(T_p\) |
| False positives | \(F_p\) |
| False negatives | \(F_n\) |
| True negatives | \(T_n\) |
| Total | \(N = T_p + F_p + F_n + T_n\) |
| Accuracy | \(\mathrm{Acc} = (T_p + T_n)/N\) |
| Precision | \(P = \frac{T_p}{T_p + F_p}\) |
| Recall | \(R = \frac{T_p}{T_p + F_n}\) |
| Specificity | \(\mathrm{TNR} = \frac{T_n}{F_p + T_n}\) |
| False positive rate | \(\mathrm{FPR} = \frac{F_p}{F_p + T_n}\) |
| \(F_1\) score | \(F_1 = \frac{2 P R}{P + R}\) |
| Logistic function | \(\sigmoidf{z} = \frac{1}{1 + \e{-z}}\) |
| Value at zero | \(\sigmoidf{0} = \frac{1}{2}\) |
| Symmetry | \(\sigmoidf{-z} = 1 - \sigmoidf{z}\) |
| Derivative | \(\sigmoid'\paren{z} = \sigmoidf{z}\paren{1 - \sigmoidf{z}}\) |
| Logistic regression model | \(\Prf{\Outvar = 1 \given \vx} = \sigmoidf{\dprod{\vcoef}{\vx}}\) |
| Odds | \(\odds = \prob / (1 - \prob)\) |
| Log odds (logit) | \(\logodds = \ln \dfrac{\prob}{1 - \prob}\) |
| Probability to log odds | \(\prob \underset{\xleftarrow[\odds / (1 + \odds)]{}}{\xrightarrow{\prob / (1 - \prob)}} \odds \underset{\xleftarrow[\e{\logodds}]{}}{\xrightarrow{\ln \odds}} \logodds\) |
| Logistic regression model, log-odds form | \(\logodds = \dprod{\vcoef}{\vx}\) |
| Odds multiplier of \(\coef{j}\) | \(\e{\coef{j}}\) |
| Threshold rule | \(\modmemf{\vx} = \indicp{\Prf{\Outvar = 1 \given \vx} > \thresh}\) |
| Decision boundary at a threshold | \(\dprod{\vcoef}{\vx} = \ln \frac{\thresh}{1 - \thresh}\) |
| Probability of one Bernoulli outcome | \(\Prf{\Outvar = y \given \vx} = \prob^{y} \paren{1 - \prob}^{1 - y}\) |
| Log loss | \(\lossf{y, \prob} = -y \ln \prob - (1 - y) \ln (1 - \prob)\) |
| Total log loss | \(C(\vcoef) = \sumin \lossf{y_i, \prob_i}\) |
| Gradient of the total log loss | \(\grad C(\vcoef) = \sumin (\prob_i - y_i)\, \vx_i\) |
| Hessian of the total log loss | \(\grad^2 C(\vcoef) = \sumin \prob_i (1 - \prob_i)\, \vx_i \tp{\vx_i}\) |
| Newton update | \(\vcoef_{t} = \vcoef_{t-1} - \inv{\grad^2 C(\vcoef_{t-1})} \grad C(\vcoef_{t-1})\) |
| Power rule | \(\dif{w^k}{w} = k w^{k-1}\) |
| Gradient | \(\grad C(\vcoef) = \tp{\sb{\derivf{C}{\coef{1}}, \ldots, \derivf{C}{\coef{p}}}}\) |
| Step size | \(h > 0\) |
| Gradient descent update | \(\vcoef_{t} = \vcoef_{t-1} - h \grad C(\vcoef_{t-1})\) |
| Arguments of the logarithm | \(a, b > 0\) |
| Log of one | \(\ln 1 = 0\) |
| Exponential of zero | \(\e{0} = 1\) |
| Log of an exponential | \(\lnf{\e{s}} = s\) |
| Exponential of a log | \(\e{\ln a} = a\) |
| Log of a reciprocal | \(\lnf{1 / a} = -\ln a\) |
| Exponential of a negative | \(\e{-s} = 1 / \e{s}\) |
| Log of a product | \(\lnf{a b} = \ln a + \ln b\) |
| Log of a quotient | \(\lnf{a / b} = \ln a - \ln b\) |
| Log of a power | \(\lnf{a^k} = k \ln a\) |
| Exponential of a sum | \(\e{s + t} = \e{s} \e{t}\) |
| Exponential of a difference | \(\e{s - t} = \e{s} / \e{t}\) |
| Power of an exponential | \(\paren{\e{s}}^k = \e{k s}\) |
| \(\euler\) | \(\e{-1}\) | \(\ln 2\) | \(\ln 3\) | \(\ln 4\) | \(\ln 5\) | \(\ln 10\) | |
|---|---|---|---|---|---|---|---|
| Value | \(2.718\) | \(0.368\) | \(0.693\) | \(1.099\) | \(1.386\) | \(1.609\) | \(2.303\) |
| Outcome | \(\Outvar = \truemeanf{x} + \noise\) |
| Noise variance | \(\Var{\noise} = \varpar\) |
| Prediction error | \(\prederr = \devof{\predval}{\Outvar}\) |
| Bias | \(\bias{\predval} = \E{\prederr}\) |
| Deviation of the prediction | \(\popdevf{\predval} = \predval - \Epredval\) |
| Variance | \(\Var{\predval} = \E{\sqf{\popdevf{\predval}}}\) |
| Expected squared error | \(\ESE{x} = \E{\sqf{\prederr}}\) |
| Expected squared error decomposition | \(\ESE{x} = \sqf{\bias{\predval}} + \Var{\predval} + \varpar\) |
| Empirical risk | \(\empriskf{\modmem; \dataset} = \avgin \lossf{y_i, \modmemf{\vx_i}}\) |
| Training error | \(\empriskf{\modmem; \trainset}\) |
| Test error | \(\empriskf{\modmem; \testset}\) |
| Cross-validation estimate, equal-size-fold case | \(\hR_{k\text{-fold}} = \frac{1}{k} \sum_{j=1}^{k} \MSEt_j\) |
| Risk | \(R(\vcoef) = \E{(\dprod{\vcoef}{\vfeatmap(\vx)} - y)^2}\) |
| Sample MSE | \(\sMSE{\vcoef} = \frac{1}{n} \sumin (\dprod{\vcoef}{\vfeatmap(\vx_i)} - y_i)^2\) |
| Sample MSE, matrix form | \(\sMSE{\vcoef} = \frac{1}{n} \norm{\design \vcoef - \vy}_2^2\) |
| Gradient of the sample MSE | \(\grad \sMSE{\vcoef} = \frac{2}{n} \tp{\design} (\design \vcoef - \vy)\) |
| Normal equations | \(\stprod{\design} \hvcoef = \tprod{\design}{\vy}\) |
| Unique solution condition | \(\detf{\stprod{\design}} \neq 0\) |
| Solution | \(\hvcoef = (\stprod{\design})^{-1} \tprod{\design}{\vy}\) |
| Fitted values | \(\est{\vy} = \design \hvcoef\) |
| Residual vector | \(\vresid = \vy - \est{\vy}\) |
| Intercept trick | \(\vfeatmap(\vx) = \tp{\sb{1, x_1, \ldots, x_{\inputdim}}}\) |
| Predictor matrix | \(\mx = \tp{\sb{\vx_1, \ldots, \vx_n}} \in \reals^{n \times \inputdim}\) |
| Design function | \(\design(\mx) = \tp{\sb{\vfeatmap(\vx_1), \ldots, \vfeatmap(\vx_n)}}\) |
| Design matrix | \(\design = \design(\mx)\) |
| Design matrix shape | \(\design \in \reals^{n \times p^*}\) |
| Coefficient vector shape | \(\vcoef \in \reals^{p^*}\) |
| Predictions | \(\est{\vy} = \design \vcoef\) |
| Feature map | \(\vfeatmap : \reals^{\inputdim} \to \reals^{p^*}\) |
| Linear model in the features | \(\modmemf{\vx} = \dprod{\vcoef}{\vfeatmap(\vx)}\) |
| Monomial | \(m(\vx) = \prod_{j=1}^{\inputdim} x_j^{a_j}\) |
| Monomial degree bound | \(\sum_{j=1}^{\inputdim} a_j \le d\) |
| Linear form | \(\deriv{\vcoef} \dprod{\vecf{c}}{\vcoef} = \vecf{c}\) |
| Squared norm | \(\deriv{\vcoef} \dprod{\vcoef}{\vcoef} = 2 \vcoef\) |
| Matrix-vector product, constant \(\matr{A}\) | \(\deriv{\vcoef} (\matr{A} \vv) = (\deriv{\vcoef} \vv) \tp{\matr{A}}\) |
| Special case \(\vv = \vcoef\) | \(\deriv{\vcoef} (\matr{A} \vcoef) = \tp{\matr{A}}\) |
| Next rule needs \(\matr{S}\) symmetric | \(\matr{S} = \tp{\matr{S}}\) |
| Quadratic form | \(\deriv{\vcoef} \qform{\matr{S}}{\vcoef} = 2 \matr{S} \vcoef\) |
| Matrix product shape | \((a \times b)(b \times c) \to a \times c\) |
| Transpose shape | \(a \times b \to b \times a\) |
| Dot product | \(\dprod{\vx}{\vy} = \sumin x_i y_i\) |
| Sample mean | \(\smean{y} = \frac{1}{n} \sumin y_i\) |
| Sum rule | \(\dif{}{w} \sumin u_i = \sumin \dif{u_i}{w}\) |
| Chain rule | \(\dif{h(u)}{w} = \dif{h(u)}{u} \, \dif{u}{w}\) |
| Minimum: first derivative is zero | \(\derivf{\hR}{w} = 0\) |
| Minimum: second derivative is positive | \(\dderivf{\hR}{w} > 0\) |