Sufficient Statistic

Recall that a Statistic is a “measurement” of the sample. A statistic is sufficient if we can “recover” the sample distribution without knowing the true parameter. Formally, A statistic $T$ is sufficient for Statistical Model $\{P_{\theta }\}$ if the conditional distribution of the sample given $T$ does not depend on $\theta$. In symbolic form, we have

$$(X\perp \perp \theta )\mid T(X)\text{or}\theta \perp \perp X\mid T(X).$$

In the language of causal inference, the statistic “blocks” the causal chain

$$\theta \to T(X)\to X.$$

Info

• The intuition is that, if I know $T$ but not $\theta$, I can simulate $X^{\prime }\sim \cdot \mid T=t$ as good as¹ $X\sim P_{\theta }$.
• A sufficient statistic is a “lossless” compression of the data that has all the information about the parameter.

Fisher-Neyman Factorization Theorem

Thm

Suppose $\mathcal{X}={\mathbb{R}}^d$, distribution $P_{\theta }$ has density $p_{\theta }$ for all $\theta \in \Theta$. Then, a statistic $T:\mathcal{X}\to {\mathbb{R}}^k$ is sufficient iff $p_{\theta }=g(T(x),\theta )\cdot f(x)$, for some functions $g:{\mathbb{R}}^k\times \Theta \to {\mathbb{R}}_{\ge 0}$ and $f:\Theta \to {\mathbb{R}}_{\ge 0}$.

• ❗️ The result also holds for PMFs.

The proof is straightforward:

$$p(x\mid T=t)=\frac{p_{\theta }(x,t)}{\int p_{\theta }(x,t)\mathrm{d}x}=\frac{g(t,\theta )\cdot f(x)\cdot \mathrm{𝟙}\{T(x)=t\}}{\int g(t,\theta )\cdot f(x)\cdot \mathrm{𝟙}\{T(x)=t\}\mathrm{d}x}\propto f(x)\mathrm{𝟙}\{T(x)=t\}\perp \perp \theta .$$

• ❗️ $f(x)$ can be a constant. Then, $g$, and thus $T$, is just a reformulation of $p_{\theta }$. See !todo homework 1.1.

Examples

• $T(X)=X$ is always a sufficient statistic.

• Order Statistics $T(X)=(X_{(1)},\dots ,X_{(n)})$ is sufficient.

• For $n$ iid Gaussian r.v.s with unit variance, $p_{\theta }=(2\pi {)}^{-n/2}\exp \left(-1/2\left(-\theta {\sum }_{i=1}^n{x}_i+n{\theta }^2+{\sum }_{i=1}^n{x}_i^2\right)\right)$. Thus, $T={\sum }_{i=1}^n{X}_i$ and $g(T,\theta )=\exp (-1/2(-\theta T+n{\theta }^2))$.

• ❗️ For iid Gaussian r.v.s with a known variance, the Order Statistics $(X_{(i)})\in {\mathbb{R}}^n$ is considered “bigger” than $\overline{X}=\frac{1}{n}{\sum }_{i=1}^n{X}_i\in \mathbb{R}$, and the latter is more “compressed”.

Gaussian Linear Model

For a fixed overdetermined ($n\ge d$) design matrix $X\in {\mathbb{R}}^{n\times d}$ and the Gaussian linear model is $Y=X\theta +ϵ$, $ϵ\sim \mathcal{N}(0,{\sigma }^2{I}_n)$. Let $U=X(X^{T}X{)}^{-1/2}$. Then, $T(Y)=U{U}^{T}Y$ is sufficient for $Y$.

One way to prove this is to express $Y$ in $T$ and show that the expression does not depend on $\theta$. We have

$$T=X(X^{T}X{)}^{-1}{X}^T(X\theta +ϵ)=X\theta +U{U}^Tϵ=Y-(I-U{U}^T)ϵ.$$

Note that $I-U{U}^T$ is a projector and $X^T(I-U{U}^T)=0$. Thus,

$$Y=T+{\mathrm{Proj}}_{\mathrm{Ker}(X^T)}ϵ.$$

Specifically,

$$Y\mid T(y)=t\sim \mathcal{N}(t,{\sigma }^{2}P)\perp \perp \theta .$$

Intuitively, the date $Y$ is high-dimensional ($n$) while the useful information $\theta$ is low-dimensional ($d$). The statistic $T$ maps the data to the column space of $X$, which is also of dimension $d$. The remaining part is pure noise, orthogonal to the column space of $X$.

See Gaussian Linear Model for more details.

Rmk

The sufficient statistic is not unique.

Rao-Blackwell Theorem

Thm

Suppose the action space $\mathcal{A}$ is convex, the loss function $L(a,\theta )$ is convex in $a$, and $T$ is a sufficient statistic for $\theta$. Then, for any statistical procedure $A$, consider

$$A^{\prime }(x)={\mathbb{E}}_{X^{\prime }\sim \cdot \mid T(x)}[A(X^{\prime })\mid T(x)].$$

We have

$$R(A^{\prime };\theta )\le R(A;\theta ),\forall \theta \in \Theta .$$

Applications

Gaussian Median

Rao-Blackwell theorem gives a better estimator for the median of a Gaussian distribution with known variance than the sample median. Since $T=\overline{X}$ is sufficient, we can do a symmetric sampling such that the new sample satisfies $\frac{1}{m}{\sum }_{j=1}^m{X}_j^{\prime }=T=\overline{X}$. Then, we use the sample median of the new sample as the estimator.

Order Should Not Matter

Rao-Blackwell theorem is useful in proving Admissibility. Suppose the loss function $L$ is strictly convex in $a$. If $A:[0,1{]}^n\to [0,1]$ is not order-invariant to its $n$ arguments, then $A$ is not admissible.

To see this, we use the Order Statistics as the sufficient statistic. Since $[0,1]$ is convex, by Rao-Blackwell theorem, we consider

$$A^{\prime }(x)=\mathbb{E}[A(X)\mid (x_{(i)})]=\frac{1}{n!}\sum _{\sigma \in S_n}A(x_{(\sigma (i))})=\frac{1}{n!}\sum _{\sigma \in S_n}A(x_{\sigma (i)}),$$

where $S_n$ is the permutation group of $n$ elements and the last equality is because the summation is over all permutations. Clearly, $A^{\prime }$ is order-invariant. Further, by the strict convexity of $L$, we have

$$\begin{array}{rl}R(A^{\prime },\theta )=& \mathbb{E}\left[L\left(\frac{1}{n!}\sum _{\sigma }A(\sigma (X)),\theta \right)\right]\\ <& \mathbb{E}\left[\frac{1}{n!}\sum _{\sigma }L\left(A(\sigma (X)),\theta \right)\right]\\ =& \frac{1}{n!}\sum _{\sigma }\mathbb{E}\left[L\left(A(\sigma (X)),\theta \right)\right]\\ =& \mathbb{E}\left[L\left(A(X),\theta \right)\right]\\ =& R(A,\theta ),\end{array}$$

where the penultimate equality is because $\sigma (X)\stackrel{d}{=}X$ due to iidness, and the strict inequality is by Jensen’s inequality; the equality holds iff $A({\sigma }_i)=A({\sigma }_j)$ for any two permutations, which implies $A$ is order-invariant.

If the loss function is Mean Squared Error, we have an alternative proof that has a better interpretation. Recall the MSE decomposition:

$$\mathrm{MSE}=\mathrm{Var}+{\mathrm{Bias}}^2.$$

Note that

$$\mathrm{Bias}(A^{\prime })=\mathbb{E}[A^{\prime }(X)-\theta ]=\frac{1}{n}\sum _{\sigma }\mathbb{E}[A(\sigma (X))-\theta ]=\mathbb{E}[A(X)-\theta ]=\mathrm{Bias}(A).$$

So we only need to compare the variance:

$$\mathrm{Var}(A^{\prime })=\mathrm{Var}\left(\frac{1}{n!}\sum _{\sigma }A\circ \sigma \right)=\frac{1}{(n!{)}^2}\mathrm{Var}\left(\sum _{\sigma }A\circ \sigma \right)\le \frac{(n!{)}^2}{(n!{)}^2}\mathrm{Var}(A)=\mathrm{Var}(A),$$

where the inequality is due to Cauchy-Schwartz Inequality, and the equality holds if and only if

$$\begin{array}{rl}& A\circ {\sigma }_i\stackrel{d}{=}cA\circ {\sigma }_j,\exists c,\forall {\sigma }_i,{\sigma }_j\in S_n\\ ⟺& A\circ {\sigma }_i\stackrel{d}{=}A\circ {\sigma }_j\\ ⟺& A\text{ is order-invariant,}\end{array}$$

where the first equivalence is because

$$\mathrm{Var}(A)=\mathrm{Var}(A\circ {\sigma }_i)=c^2\mathrm{Var}(A\circ {\sigma }_j)=c^2\mathrm{Var}(A)⟹c=1.$$

Footnotes

1. Equal in distribution. ↩