Midterm 2 note sheet

Download the PDF

Table 1: Performance metrics.
Confusion matrix rows: the true class \(Y\)
Confusion matrix columns: the predicted class \(\hat Y\)
True positives \(T_p\)
False positives \(F_p\)
False negatives \(F_n\)
True negatives \(T_n\)
Total \(N = T_p + F_p + F_n + T_n\)
Accuracy \(\mathrm{Acc} = (T_p + T_n)/N\)
Precision \(P = \frac{T_p}{T_p + F_p}\)
Recall \(R = \frac{T_p}{T_p + F_n}\)
Specificity \(\mathrm{TNR} = \frac{T_n}{F_p + T_n}\)
False positive rate \(\mathrm{FPR} = \frac{F_p}{F_p + T_n}\)
\(F_1\) score \(F_1 = \frac{2 P R}{P + R}\)
Table 2: Logistic regression.
Logistic function \(\sigmoidf{z} = \frac{1}{1 + \e{-z}}\)
Value at zero \(\sigmoidf{0} = \frac{1}{2}\)
Symmetry \(\sigmoidf{-z} = 1 - \sigmoidf{z}\)
Derivative \(\sigmoid'\paren{z} = \sigmoidf{z}\paren{1 - \sigmoidf{z}}\)
Logistic regression model \(\Prf{\Outvar = 1 \given \vx} = \sigmoidf{\dprod{\vcoef}{\vx}}\)
Odds \(\odds = \prob / (1 - \prob)\)
Log odds (logit) \(\logodds = \ln \dfrac{\prob}{1 - \prob}\)
Probability to log odds \(\prob \underset{\xleftarrow[\odds / (1 + \odds)]{}}{\xrightarrow{\prob / (1 - \prob)}} \odds \underset{\xleftarrow[\e{\logodds}]{}}{\xrightarrow{\ln \odds}} \logodds\)
Logistic regression model, log-odds form \(\logodds = \dprod{\vcoef}{\vx}\)
Odds multiplier of \(\coef{j}\) \(\e{\coef{j}}\)
Threshold rule \(\modmemf{\vx} = \indicp{\Prf{\Outvar = 1 \given \vx} > \thresh}\)
Decision boundary at a threshold \(\dprod{\vcoef}{\vx} = \ln \frac{\thresh}{1 - \thresh}\)
Probability of one Bernoulli outcome \(\Prf{\Outvar = y \given \vx} = \prob^{y} \paren{1 - \prob}^{1 - y}\)
Log loss \(\lossf{y, \prob} = -y \ln \prob - (1 - y) \ln (1 - \prob)\)
Total log loss \(C(\vcoef) = \sumin \lossf{y_i, \prob_i}\)
Gradient of the total log loss \(\grad C(\vcoef) = \sumin (\prob_i - y_i)\, \vx_i\)
Hessian of the total log loss \(\grad^2 C(\vcoef) = \sumin \prob_i (1 - \prob_i)\, \vx_i \tp{\vx_i}\)
Newton update \(\vcoef_{t} = \vcoef_{t-1} - \inv{\grad^2 C(\vcoef_{t-1})} \grad C(\vcoef_{t-1})\)
Table 3: Gradient descent.
Power rule \(\dif{w^k}{w} = k w^{k-1}\)
Gradient \(\grad C(\vcoef) = \tp{\sb{\derivf{C}{\coef{1}}, \ldots, \derivf{C}{\coef{p}}}}\)
Step size \(h > 0\)
Gradient descent update \(\vcoef_{t} = \vcoef_{t-1} - h \grad C(\vcoef_{t-1})\)
Table 4: Logarithms and exponentials.
Arguments of the logarithm \(a, b > 0\)
Log of one \(\ln 1 = 0\)
Exponential of zero \(\e{0} = 1\)
Log of an exponential \(\lnf{\e{s}} = s\)
Exponential of a log \(\e{\ln a} = a\)
Log of a reciprocal \(\lnf{1 / a} = -\ln a\)
Exponential of a negative \(\e{-s} = 1 / \e{s}\)
Log of a product \(\lnf{a b} = \ln a + \ln b\)
Log of a quotient \(\lnf{a / b} = \ln a - \ln b\)
Log of a power \(\lnf{a^k} = k \ln a\)
Exponential of a sum \(\e{s + t} = \e{s} \e{t}\)
Exponential of a difference \(\e{s - t} = \e{s} / \e{t}\)
Power of an exponential \(\paren{\e{s}}^k = \e{k s}\)
Table 5: Decimal values of exponentials and logarithms.
\(\euler\) \(\e{-1}\) \(\ln 2\) \(\ln 3\) \(\ln 4\) \(\ln 5\) \(\ln 10\)
Value \(2.718\) \(0.368\) \(0.693\) \(1.099\) \(1.386\) \(1.609\) \(2.303\)
Table 6: Generalization.
Outcome \(\Outvar = \truemeanf{x} + \noise\)
Noise variance \(\Var{\noise} = \varpar\)
Prediction error \(\prederr = \devof{\predval}{\Outvar}\)
Bias \(\bias{\predval} = \E{\prederr}\)
Deviation of the prediction \(\popdevf{\predval} = \predval - \Epredval\)
Variance \(\Var{\predval} = \E{\sqf{\popdevf{\predval}}}\)
Expected squared error \(\ESE{x} = \E{\sqf{\prederr}}\)
Expected squared error decomposition \(\ESE{x} = \sqf{\bias{\predval}} + \Var{\predval} + \varpar\)
Empirical risk \(\empriskf{\modmem; \dataset} = \avgin \lossf{y_i, \modmemf{\vx_i}}\)
Training error \(\empriskf{\modmem; \trainset}\)
Test error \(\empriskf{\modmem; \testset}\)
Cross-validation estimate, equal-size-fold case \(\hR_{k\text{-fold}} = \frac{1}{k} \sum_{j=1}^{k} \MSEt_j\)
Table 7: Linear regression.
Risk \(R(\vcoef) = \E{(\dprod{\vcoef}{\vfeatmap(\vx)} - y)^2}\)
Sample MSE \(\sMSE{\vcoef} = \frac{1}{n} \sumin (\dprod{\vcoef}{\vfeatmap(\vx_i)} - y_i)^2\)
Sample MSE, matrix form \(\sMSE{\vcoef} = \frac{1}{n} \norm{\design \vcoef - \vy}_2^2\)
Gradient of the sample MSE \(\grad \sMSE{\vcoef} = \frac{2}{n} \tp{\design} (\design \vcoef - \vy)\)
Normal equations \(\stprod{\design} \hvcoef = \tprod{\design}{\vy}\)
Unique solution condition \(\detf{\stprod{\design}} \neq 0\)
Solution \(\hvcoef = (\stprod{\design})^{-1} \tprod{\design}{\vy}\)
Fitted values \(\est{\vy} = \design \hvcoef\)
Residual vector \(\vresid = \vy - \est{\vy}\)
Table 8: Design matrix.
Intercept trick \(\vfeatmap(\vx) = \tp{\sb{1, x_1, \ldots, x_{\inputdim}}}\)
Predictor matrix \(\mx = \tp{\sb{\vx_1, \ldots, \vx_n}} \in \reals^{n \times \inputdim}\)
Design function \(\design(\mx) = \tp{\sb{\vfeatmap(\vx_1), \ldots, \vfeatmap(\vx_n)}}\)
Design matrix \(\design = \design(\mx)\)
Design matrix shape \(\design \in \reals^{n \times p^*}\)
Coefficient vector shape \(\vcoef \in \reals^{p^*}\)
Predictions \(\est{\vy} = \design \vcoef\)
Table 9: Feature expansion.
Feature map \(\vfeatmap : \reals^{\inputdim} \to \reals^{p^*}\)
Linear model in the features \(\modmemf{\vx} = \dprod{\vcoef}{\vfeatmap(\vx)}\)
Monomial \(m(\vx) = \prod_{j=1}^{\inputdim} x_j^{a_j}\)
Monomial degree bound \(\sum_{j=1}^{\inputdim} a_j \le d\)
Table 10: Vector calculus.
Linear form \(\deriv{\vcoef} \dprod{\vecf{c}}{\vcoef} = \vecf{c}\)
Squared norm \(\deriv{\vcoef} \dprod{\vcoef}{\vcoef} = 2 \vcoef\)
Matrix-vector product, constant \(\matr{A}\) \(\deriv{\vcoef} (\matr{A} \vv) = (\deriv{\vcoef} \vv) \tp{\matr{A}}\)
Special case \(\vv = \vcoef\) \(\deriv{\vcoef} (\matr{A} \vcoef) = \tp{\matr{A}}\)
Next rule needs \(\matr{S}\) symmetric \(\matr{S} = \tp{\matr{S}}\)
Quadratic form \(\deriv{\vcoef} \qform{\matr{S}}{\vcoef} = 2 \matr{S} \vcoef\)
Table 11: Shapes and calculus.
Matrix product shape \((a \times b)(b \times c) \to a \times c\)
Transpose shape \(a \times b \to b \times a\)
Dot product \(\dprod{\vx}{\vy} = \sumin x_i y_i\)
Sample mean \(\smean{y} = \frac{1}{n} \sumin y_i\)
Sum rule \(\dif{}{w} \sumin u_i = \sumin \dif{u_i}{w}\)
Chain rule \(\dif{h(u)}{w} = \dif{h(u)}{u} \, \dif{u}{w}\)
Minimum: first derivative is zero \(\derivf{\hR}{w} = 0\)
Minimum: second derivative is positive \(\dderivf{\hR}{w} > 0\)
Back to top