Anderson’s Lemma

Anderson

Given a Bowl-Shaped Loss $\ell$ and a Guassian noise $ϵ\sim \mathcal{N}(0,\Sigma )$, we have

$$R_1(\theta ):=\mathbb{E}\ell (\theta +ϵ)$$

is minimized at $\theta =0$. This is saying that for any distribution $P_{\theta }$ of $\theta$, we have

$$\int \ell \mathrm{d}\mathcal{N}\le \int \ell \mathrm{d}(\mathcal{N}\ast P_{\theta }),$$

with the equality holding with $P_{\theta }={\delta }_0$.

Anderson’s lemma gives the following famous result:

Cor

The Bayes Optimal Estimator w.r.t a Bowl-Shaped Loss and a Gaussian prior posterior is the posterior mean.

Pf

We know that the posterior distribution of the parameter $\theta \mid X$ is Gaussian given a Gaussian prior. Thus, the posterior risk is

$${\mathbb{E}}_{\theta \mid X}\ell (a-\theta )={\mathbb{E}}_{ϵ}\ell (a-{\hat{\theta }}_{\mathrm{post}}^{\mathrm{mean}}+ϵ),$$

where $ϵ\sim \mathcal{N}(0,{\hat{\Sigma }}_{\mathrm{post}})$. Therefore, by Anderson’s lemma, the posterior risk is minimized at ${\hat{\theta }}_{\mathrm{post}}^{\mathrm{mean}}$, which is hence the Bayes Optimal Estimator.