Evaluating an Estimator

• ❗️ This note focuses on point estimators.

As with any statistical decision-making problem, the first step in an estimation task is to define the performance metrics of the estimator. Unlike hypothesis testing, where its binary nature gives simple metrics involving only Type I and II errors, evaluating an estimator involves more considerations. We start by distinguishing the probabilistic and statistical properties of an estimator.

An estimator of the parameter $\theta$ is conventionally denoted as ${\hat{\theta }}_X$ to emphasize its dependence on the sample $X\sim P_{\theta }^m$. And note that the sample further depends on the underlying parameter $\theta$ and the dimension (number of samples) $m$. So we also sometimes denote the estimator as ${\hat{\theta }}_m$.

• Probabilistic properties investigate the expected behavior of the estimator w.r.t. the parameter $\theta$.
• Statistical properties investigate the asymptotic or non-asymptotic behavior of the estimator as the number of samples $m$ goes to infinity.

For example, although both measuring the estimator’s “distance” to the true parameter, Bias, a probabilistic property, and Consistency, a statistical property, provide different insights into the estimator’s performance.

This note discusses the following metrics

• Probabilistic properties
  • Bias
  • Standard Error
  • Risk
• Statistical properties
  • Consistency
  • Asymptotic Normality

Bias

The bias of an estimator is

$$\mathrm{bias}({\hat{\theta }}_m)=\mathbb{E}({\hat{\theta }}_m)-\theta$$

where the expectation is over the data generating distribution which is determined by the true parameter $\theta$.

• An estimator ${\hat{\theta }}_m$ is said to be unbiased if $\mathrm{bias}({\hat{\theta }}_m)=0,$ which implies that $\mathbb{E}({\hat{\theta }}_m)=\theta$.

• An estimator ${\hat{\theta }}_m$ is said to be asymptotically unbiased if ${\lim }_{m\to \infty }\mathrm{bias}({\hat{\theta }}_m)=0$.

  • ❗️ Asymptotic unbiasedness is NOT equivalent to consistency.

Example: Variance Estimator

We compare two different estimators of the variance parameter ${\sigma }^2$ of a Normal Distribution.

Sample Variance

$${\hat{\sigma }}_m^2=\frac{1}{m}\sum _{i=1}^m(x^{(i)}-{\hat{\mu }}_m{)}^2$$

where ${\hat{\mu }}_m$ is the sample mean. Then the bias of the variance estimator is

$$\begin{array}{rl}\mathbb{E}[{\hat{\sigma }}_m^2]& =\frac{1}{m}\sum \mathbb{E}[(x^{(i)}-\mu +\mu -{\hat{\mu }}_m{)}^2]\\ & ={\sigma }^2-\frac{1}{m}\left(2\mathbb{E}[(\mu -{\hat{\mu }}_m)(\sum x^{(i)}-m\mu )]\right)+\mathbb{E}[(\mu -{\hat{\mu }}_m{)}^2]\\ & ={\sigma }^2-\mathrm{Var}({\hat{\mu }}_m)\end{array}$$

Here from the i.i.d condition we have

$$\begin{array}{rl}\mathrm{Var}({\hat{\mu }}_m)& ={\mu }^2-2\mu \mathbb{E}[{\hat{\mu }}_m]+\frac{1}{m}\mathbb{E}[(x^{(1)}{)}^2]+\frac{m-1}{m}{\mu }^2\\ & =\frac{1}{m}(\mathbb{E}[(x^{(1)}{)}^2]-{\mu }^2)\\ & =\frac{{\sigma }^2}{m}\end{array}$$

Thus we get

$$\mathbb{E}[{\hat{\sigma }}_m]=\frac{m-1}{m}{\sigma }^2$$

So the bias of sample variance estimator is $-\frac{1}{m}{\sigma }^2$, hence it is a biased estimator

Unbiased Sample Variance

$${\tilde{\sigma }}_m^2=\frac{1}{m-1}\sum _{i=1}^m(x^{(i)}-{\hat{\mu }}_m{)}^2=\frac{m}{m-1}{\hat{\sigma }}_m^2$$

From the calculation above, we know that unbiased sample variance estimator ${\tilde{\sigma }}_m^2$ is unbiased.

Standard Error

The standard error is

$$\mathrm{SE}(\hat{\theta })=\sqrt{\mathrm{Var}(\hat{\theta })}$$

where the variance is taken over the data generating distribution which is determined by the true parameter $\theta$.

Unbiasedness has no constancy

• Neither the square root of the sample variance nor the square root of the unbiased estimator of the variance provide an unbiased estimate of the standard error
  • Both approaches tend to underestimate the true standard error, but are still used in practice.
  • The square root of the unbiased estimator of the variance is less of an underestimate. For large $m$, the approximation is quite reasonable
• In general, the unbiasedness is not preserved under the transformation

Example: Error Mean

Using $\hat{\mu }$ to denote the mean of the data, from the calculation in Example Variance Estimator, we have

$$\mathrm{SE}(\hat{\mu })=\frac{\sigma }{\sqrt{m}}$$

where $\sigma$ is the true standard deviation of the data, and $m$ is the number of data points.

Standard error is often combined with CLT to constrict Confidence Intervals.

For example, in machine learning experiments, we often estimate the generalization error by computing the sample mean of the error on the test set. The number of examples in the test set determines the accuracy of this estimate. Taking advantage of the Central Limit Theorem, which tells us that the mean will be approximately distributed with a normal distribution, we can use the standard error to compute the probability that the true expectation falls in any chosen interval. For example, the 95% Confidence Interval centered on the mean ${\hat{\mu }}_m$ is

$$({\hat{\mu }}_m-1.96\mathrm{SE}({\hat{\mu }}_m),{\hat{\mu }}_m+1.96\mathrm{SE}({\hat{\mu }}_m))$$

under the normal distribution with mean ${\hat{\mu }}_m$ and variance $\mathrm{SE}({\hat{\mu }}_m{)}^2$. In machine learning experiments, it is common to say that algorithm A is better than algorithm B if the upper bound of the 95% confidence interval for the error of algorithm A is lower than the lower bound of the 95% confidence interval for the error of algorithm B.

Risk

As seen in previous sections, we can have different probabilistic (expected) performance metric of an estimator. Risk is the most general probabilistic metric as it can incorporate any loss function.

For example, Mean Squared Error is the quadratic risk using a quadratic loss function $L(\hat{\theta },\theta )=(\hat{\theta }-\theta {)}^2$; and it encompasses both Bias and Standard Error: $\mathrm{MSE}={\mathrm{Bias}}^2+{\mathrm{SE}}^2$.

Further, if we want to control the risk across all possible values of the parameter $\theta$, we can consider optimality in terms of Bayes risk and minimax risk.

Consistency

We wish that as the number of data in our dataset increases, our point estimates converge in probability to the true value. This condition is called consistency:

$${\hat{\theta }}_m\stackrel{P}{⟶}\theta$$

Consistency vs Asymptotic Unbiasedness

$$\text{Consistency}\begin{array}{r}⟹\\ /⟸\end{array}\text{Asymptotic Unbiasedness}$$

Thm

If $\mathrm{bias}({\hat{\theta }}_n)\to 0$ and $\mathrm{SE}({\hat{\theta }}_n)\to 0$, then ${\hat{\theta }}_n$ is a consistant estimator.

The above can be proved using Chebyshev Inequality or the Relation between Convergence Modes ($L_2$ convergence implies convergence in probability).

Asymptotic Normality

We wish that as the number of data in our dataset increases, our point estimates behave like following a normal distribution. This condition is called asymptotic normality:

$$\sqrt{m}({\hat{\theta }}_m-\theta )\stackrel{d}{⟶}\mathcal{N}(0,\Sigma )$$

Asymptotic normality tells us that the deviation of the estimator from the true parameter has an exponentially decaying tail bound, and thus we can construct tight Confidence Intervals around the estimator.

For an asymptotic normal estimator, its statistical difficulty¹ is governed by its asymptotic variance $\Sigma$.

Footnotes

1. Or statistical complexity/efficiency. See also Best Statistical Efficiency. ↩