Bayes Optimal Estimator

Given a loss function $L:\mathcal{A}\times \Theta \to {\mathbb{R}}_{+}$ and a prior, the Bayes optimal estimator, often shortened to Bayes estimator, minimizes the Bayes risk:

$$A^{\ast }=\arg \underset{A}{\inf }{R}_B(A)=\arg \underset{A}{\inf }{\mathbb{E}}_{\theta \sim Q}[{\mathbb{E}}_{X\sim P_{\theta }}L(A(X),\theta )].$$

Due to the Greedy principle, we have

$$A^{\ast }(x)=\arg \underset{\hat{\theta }}{\inf }{\mathbb{E}}_{\theta }[L(\hat{\theta },\theta )\mid X=x].$$

Link to original

This can also be interpreted as a Bayesian approach:

1. Have a prior over $\Theta$;
2. Observe $x$ and form the posterior $\theta \mid X=x$;
3. Act optimally according to the posterior.

Uniqueness

The Bayes estimator may not be unique. However, we have the following result.

Suppose the action space $\mathcal{A}$¹ is convex, $L(\cdot ,\theta )$ is strictly convex in the first argument for any $\theta \in \Theta$, and for any $x\in \mathcal{X}$, there exists $a\in \mathcal{A}$ such that $\mathbb{E}[L(a,\theta )\mid X=x]<\infty$. Then, the Bayes estimator is unique.

Proof of Uniqueness

Since $L$ is strictly convex, we have

$$F_x(a):=\mathbb{E}[L(a,\theta )\mid X=x],$$

is also strictly convex due to the linearity of expectation, and is proper because of the assumption. Therefore,

$$A^{\ast }(x)=\arg \underset{a\in \mathcal{A}}{\inf }{F}_x(a)$$

is uniquely defined.

We now need to show that all Bayes optimal estimator is the minimizer of $F_x$. Suppose $A^{\prime }$ is another Bayes optimal estimator. By definition,

$$R_B(A^{\prime })\le R_B(A^{\ast })⟹{\mathbb{E}}_x\left[F_x(A^{\prime }(x))-F_x(A^{\ast }(x))\right]\le 0.$$

However, by the definition of $A^{\ast }$, $F_x(A^{\prime }(x))-F_x(A^{\ast }(x))$ is a non-negative random variable. Therefore, we must have

$$F_x(A^{\prime }(x))\stackrel{\text{a.s.}}{=}{F}_x(A^{\ast }(x))=\underset{a\in \mathcal{A}}{\inf }{F}_x(a).$$

By the uniqueness of the minimizer of $F_x$, we have

$$A^{\prime }(x)=A^{\ast }(x),\text{a.s.}$$

Minimizing Posterior Risk

After calculating the posterior (see Bayesian Inference for discussion on the calculation of posterior), the next question is how to calculate the Bayes estimator, i.e., find the minimizer of the posterior risk.

For certain loss functions, their posterior risk minimizers are common functional of the posterior:

• Posterior median: $L(a,\theta )=|a-\theta |$ implies $A^{\ast }(x)=F_{\theta \mid x}^{-1}(1/2)$.
• Posterior mode: $L(a,\theta )=\mathbb{I}(a\ne \theta )$ implies $A^{\ast }(x)=\arg {\max }_{\theta }f(\theta \mid x)$.

More often, the posterior mean happens to be the Bayes estimator:

$$A^{\ast }(X)=\mathbb{E}[\theta \mid X].$$

This is the case when

• $L$ is the Mean Squared Error; or
• $L$ is a Bowl-Shaped Loss and the posterior is Gaussian.

For the first case, see Optimal Estimation. The second case is given by the Anderson’s Lemma.

Example: Gaussian Mean

Consider the Mean Squared Error. Suppose $X_i\sim \mathcal{N}(\theta ,{\sigma }^2)$ and $Q=\mathcal{N}({\mu }_0,{\tau }^2)$. The posterior is also Gaussian:

$$\theta \mid \overline{X}\sim \mathcal{N}((1-B)\overline{X}+B{\mu }_0,(1-B){\sigma }^2/n),$$

where

$$B=\frac{{\sigma }^2/n}{{\sigma }^2/n+{\tau }^2}.$$

We can see that the posterior mean is a convex combination of the prior mean and the sample mean, with $B$ being the proportion, which increases with more information in the prior ($\tau$ decreases), and decreases with more information in the sample ($n$ increases).

Thus, the Bayes optimal estimator is

$$A^{\ast }(X)=(1-B)\overline{X}+B{\mu }_0.$$

When ${\mu }_0=0$ and ${\tau }^2={\sigma }^2/n$, we have $A^{\ast }(X)=\overline{X}/2$, i.e., a regularized sample mean.

The following figure plots the risk (MSE with ${\sigma }^2=1$) for different estimators. We can see that, unlike sample mean or sample median, which have a constant risk regardless of $\theta$, the regularized sample mean has a lower risk when $\theta$ is small, but underestimate when $\theta$ is large. Therefore, if the prior puts more belief on small $\theta$, the regularized sample mean has a smaller Bayes risk.

Risk of different estimators.

Footnotes

1. Note that $A:\mathcal{X}\to \mathcal{A}$. Usually $\mathcal{A}=\Theta$ for an estimation task. ↩