Statistical Functional

Similar to a Statistic, a statistical functional is some measurement of the underlying distribution. With a slight abuse of notation, we write it as

$$T:\Delta (\mathcal{X})\to {\mathbb{R}}^d.$$

Reduction to Statistic

A Statistic is a special statistical functional evaluated at the empirical distribution, i.e., the uniform distribution over all samples:

$$T(X)=T\left(\frac{1}{n}\sum _{i=1}^n{\delta }_{X_i}\right)=:T(\hat{P}).$$

Since all information about the sample is included in its empirical distribution, the above equation is one-to-one. Thus we use the notation for Statistic $T(X)$ and statistical functional at empirical distribution $T(\hat{P})$ interchangeably.

Plug-In Principle

In general, $T(\hat{P})$ is a good estimator of $T(P)$.

Examples

• Mean. $T(P)={\int }_{\mathcal{X}}x\mathrm{d}P(x)$; the reduced statistic is the sample mean $T(\hat{P})=\frac{1}{n}{\sum }_{i=1}^n{X}_i$.
• Variance. $T(P)={\int }_{\mathcal{X}}{x}^2\mathrm{d}P(x)-({\int }_{\mathcal{X}}x\mathrm{d}P(x){)}^2$; the reduced statistic is the sample variance $T(\hat{P})=\frac{1}{n}{\sum }_{i=1}^n(X_i-\overline{X}{)}^2$.
• Least squares. $T(P)={\mathrm{argmin}}_{\Theta }{\mathbb{E}}_P(Y-{\theta }^{T}X{)}^2$; the reduced statistic is the Ordinary Least Squares $T(\hat{P})={\mathrm{argmin}}_{\Theta }\frac{1}{n}{\sum }_{i=1}^n(Y_i-{\theta }^T{X}_i{)}^2$.
• Quantiles and Order Statistics. $T(P)=F_P^{-1}(q)$, where $F_P$ is the CDF of $P$ and $q\in [0,1]$; the $k$-th Order Statistics is reduced by choosing $q=k/n$ and then $X_{(k)}=T(\hat{P})=F_n^{-1}(k/n)$.

Linear Statistical Functional

A statistical functional is called linear, if there exists a function $r:\mathcal{X}\to {\mathbb{R}}^d$ such that

$$T(P)={\int }_{\mathcal{X}}r(x)\mathrm{d}P(x).$$

• 📗 Mean is a linear statistical functional with $r(x)=x$.

Due to the linearity of integral, an implication is

$$T(a{P}_1+b{P}_2)=aT(P_1)+bT(P_2).$$

Additionally, the plug-in estimator for a linear statistical functional becomes

$$T(\hat{P})={\int }_{\mathcal{X}}r(x)\mathrm{d}\hat{P}(x)=\frac{1}{n}\sum _{i=1}^{n}r(X_i).$$

A corollary is

Corollary

Suppose a statistical functional can be expressed as $T=h(T_1,\dots ,T_K)$, where $T_k$ is a linear statistical functional for $1\le k\le K$. Then, a plug-in estimator of $T(P)$ is $h(T_1(\hat{P}),\dots ,T_K(\hat{P}))$.

This corollary covers many common statistical functionals.

Variance

Let $T_1$ and $T_2$ be the first and second moments respectively. Then, the variance is $T=T_2-T_1^2$. The corresponding plug-in estimator is $T(\hat{P})=T_2(\hat{P})-T_1(\hat{P}{)}^2=\frac{1}{n}{\sum }_{i=1}^n(X_i{)}^2-(\frac{1}{n}{\sum }_{i=1}^n{X}_i{)}^2=\frac{1}{n}{\sum }_{i=1}^n(X_i-\overline{X}{)}^2$, the sample variance.

Correlation

Let $T_{a,b}$ be the mixed moment of $X$ and $Y$ of order $a+b$. Then the correlation is

$$T=\frac{T_{1,1}-T_{1,0}{T}_{0,1}}{\sqrt{T_{2,0}-T_{1,0}^2}\sqrt{T_{0,2}-T_{0,1}^2}}=\frac{\mathbb{E}XY-\mathbb{E}X\mathbb{E}Y}{\sqrt{\mathbb{E}{X}^2-(\mathbb{E}X{)}^2}\sqrt{\mathbb{E}{Y}^2-(\mathbb{E}Y{)}^2}}.$$

Hence, the corresponding plug-in estimator is

$$T(\hat{P})=\frac{T_{1,1}(\hat{P})-T_{1,0}(\hat{P})T_{0,1}(\hat{P})}{\sqrt{T_{2,0}(\hat{P})-T_{1,0}(\hat{P}{)}^2}\sqrt{T_{0,2}(\hat{P})-T_{0,1}(\hat{P}{)}^2}}=\frac{{\sum }_{i=1}^n(X_i-\overline{X})(Y_i-\overline{Y})}{\sqrt{{\sum }_{i=1}^n(X_i-\overline{X}{)}^2}\sqrt{{\sum }_{i=1}^n(Y_i-\overline{Y}{)}^2}},$$

which is the sample correlation.