Bias-Variance Trade-Off

"20210414231119"

• An illustration of the relationship between bias, variance, Overfitting and Underfitting.

For Linear Regression

For a weight estimation $\hat{w}$, we want to evaluate the error of new estimates. Specifically, let $X$ be the training data and $x_0$ be a new observation with a real target $y_0$. We want to evaluate

$$\mathbb{E}[(y_0-x_0^T\hat{w}{)}^2|X,x_0,w]={\int }_{\mathbb{R}\times {\mathbb{R}}^n}(y_0-x_0^T\hat{w}(y){)}^{2}p(y|X,w)p(y_0|x_0,w)\mathrm{d}y\mathrm{d}{y}_0,$$

Assume $Y\sim \mathcal{N}(Xw,{\sigma }^{2}I),y_0\sim \mathcal{N}(w^T{x}_0,{\sigma }^2)$,

$$\begin{array}{rl}\mathbb{E}[(y_0-x_0^T\hat{w}{)}^2]& =\mathbb{E}{y}_0^2-2x_0^T\mathbb{E}[y_0\hat{w}]+x_0^T\mathbb{E}[\hat{w}{\hat{w}}^T]x_0\\ & =\left({\sigma }^2+(x_0^{T}w{)}^2\right)-2x_0^T\mathbb{E}[y_0]\mathbb{E}[\hat{w}]+x_0^T\left(\mathrm{Var}(\hat{w})+\mathbb{E}[\hat{w}]\mathbb{E}[\hat{w}{]}^T\right)x_0\\ & ={\sigma }^2+x_0^T\mathrm{Var}(\hat{w})x_0+(x_0^T(w-\mathbb{E}[\hat{w}]){)}^2.\end{array}$$

We see this generation error decomposes into 

1. Measurement error ${\sigma }^2$
2. Squared model bias $(x_0^T(w-\mathbb{E}[\hat{w}]){)}^2$: how close to the real solution (expected)
3. Model variance: $x_0^T\mathrm{Var}[\hat{w}]x_0$: how sensitive the estimation is

For Maximum Likelihood Estimation, the model bias is zero, but the variance if high; for Ridge Regression, the variance is reduced, though it is a biased estimator.

General Regression

For model $y=f(x;w)+ϵ,\mathbb{E}[ϵ]=0,\mathrm{Var}(ϵ)={\sigma }^2$ and the approximator $\hat{f}$, the expected error for new data is

$$\begin{array}{rl}\mathbb{E}[(y_0-\hat{f}(x_0){)}^2]& =\mathbb{E}[y_0^2]-2\mathbb{E}[y_0\hat{f}(x_0)]+\mathbb{E}[{\hat{f}}^2(x_0)]\\ & ={\sigma }^2+f_0^2-2f_0\mathbb{E}[\hat{f}(x_0)]+\mathbb{E}[\hat{f}(x_0){]}^2+\mathrm{Var}(\hat{f}(x_0))\\ & =\underset{\text{noise}}{\underbrace{{\sigma }^2}}+\underset{\text{squared bias}}{\underbrace{(f_0-\mathbb{E}[\hat{f}(x_0)]{)}^2}}+\underset{\text{variance}}{\underbrace{\mathrm{Var}(\hat{f}(x_0))}}\end{array}$$

In Other Statistical Tasks

Bias-variance trade-off was originally formulated in regression tasks. But it’s also prevalent in other statistical decision-making tasks, especially when using quadratic Risk. For example, the same analysis applies to Estimation with Mean Squared Error.

Trade-Off Negotiation

Two common ways to negotiate the trade-off between two:

• Cross-Validation
• Mean Squared Error