Central Limit Theorem

Let $\{X_i{\}}_{i=1}^n$ be a set of i.i.d. Random Variables with Mean $\mu$ and Variance ${\sigma }^2<+\infty$. Denote $\overline{X}={\sum }_{i=1}^n{X}_i/n$. Then

$$\frac{\overline{X}-\mu }{\sigma /\sqrt{n}}\stackrel{d}{⟶}Z\sim \mathcal{N}(0,1),$$

where $\stackrel{d}{\to }$ means convergence in probability.

• 💡 Also holds for multi-variate distributions: $\sqrt{n}(\overline{X}-\mu )\stackrel{d}{\to }\mathcal{N}(0,\Sigma )$.

CLT and LLN

Central limit theorem (CLT) implies the Weak Law of Large Numbers. To see this, we can rewrite the CLT as $\overline{X}\stackrel{d}{\to }\mathcal{N}(\mu ,{\sigma }^2/n)\to \mu$. That is, $\overline{X}$ converges in distribution to a point mass at $\mu$, which is equivalent to convergence in probability to $\mu$.

A non-asymptotic version of CLT, the Berry-Essèen Theorem, together with an additional bounded third moment assumption, implies the Strong Law of Large Numbers.

However, neither weak or strong LLN requires assumptions beyond finite mean. Therefore, LLN holds under weaker conditions, but offers less information about the asymptotic dynamics of the sample mean. See also ^519975.

Proof

If $X_i$ has MGF, we can use MGF to prove the theorem. To be more general, we use Characteristic Function. WLOG, we can assume $\mu =0,\sigma =1$. Let $\varphi (t)$ be the characteristic function of $X_i$. Then we have

$$\varphi (0)=1,{\varphi }^{\prime }(0)=i\mathbb{E}[X_i]=0,{\varphi }^{\prime \prime }(0)=-\mathbb{E}[X_i^2]=-1.$$

Therefore,

$$\underset{t\to 0}{\lim }\frac{\varphi (t)-\left(1-\frac{1}{2}{t}^2\right)}{t^2}=\underset{t\to \infty }{\lim }\frac{1-\frac{1}{2}{t}^2+o(t^2)-\left(1-\frac{1}{2}{t}^2\right)}{t^2}=0.$$

And the characteristic function of $\sqrt{n}\overline{X}$ is

$${\varphi }_{\overline{X}}(t)=\mathbb{E}[e^{it\sqrt{n}\overline{X}}]=\prod _{i=1}^n\mathbb{E}[e^{it/\sqrt{n}{X}_i}]=\varphi {\left(\frac{t}{\sqrt{n}}\right)}^n.$$

Then we have

$$\underset{n\to \infty }{\lim }\left|\varphi {\left(\frac{t}{\sqrt{n}}\right)}^n-{\left(1-\frac{t^2}{2n}\right)}^n\right|\le \underset{t\to \infty }{\lim }n\left|\varphi \left(\frac{t}{\sqrt{n}}\right)-\left(1-\frac{t^2}{2n}\right)\right|=\underset{n\to \infty }{\lim }{t}^2\cdot 0=0.$$

Therefore,

$$\underset{n\to \infty }{\lim }{\varphi }_{\overline{X}}(t)=\underset{n\to \infty }{\lim }{\left(1-\frac{t^2}{2n}\right)}^n=e^{-\frac{1}{2}{t}^2},$$

which is the CF of standard Normal Distribution. By the inverse property and Convergence of Characteristic Functions,

$$\sqrt{n}\overline{X}\stackrel{d}{⟶}Z\sim \mathcal{N}(0,1).$$

Sup-Norm Approximation Error: Berry-Essèen Theorem

The Berry-Essèen theorem gives a non-asymptotic bound on the difference between the CDF of the sample mean and standard normal, capturing the convergence rate in the CLT. Suppose $\mathbb{E}|X_1{|}^3<+\infty$. Let $Z_n=\sqrt{n}(\overline{X}-\mu )/\sigma$. Then

$$\underset{t}{\sup }\left|P(Z_n\le t)-\Phi (t)\right|\le \frac{33}{4}\frac{\mathbb{E}|X_1-\mu {|}^3}{\sqrt{n}{\sigma }^3}.$$

We also have a non-uniform Berry-Essèen bound, which is tighter for larger $t$:¹

$$\left|P(Z_n\le t)-\Phi (t)\right|\le c\frac{\mathbb{E}|X_1-\mu {|}^3}{\sqrt{n}{\sigma }^3(1+|t{|}^3)},$$

where $c$ is some constant². Under this additional bounded third moment assumption, this bound implies the Strong Law of Large Numbers. To see this, we have

$$P\left(|Z_n|>\sqrt{n}ϵ/\sigma \right)\le 2\Phi (-\sqrt{n}ϵ/\sigma )+2c\frac{\mathbb{E}|X_1-\mu {|}^3}{\sqrt{n}{\sigma }^3(1+n^{3/2}{ϵ}^3/{\sigma }^3)},$$

which implies

$$\sum _{n=1}^{\infty }P(|{\overline{X}}_n-\mu |>ϵ)\lesssim \sum _{n=1}^{\infty }\Phi (-\sqrt{n}ϵ/\sigma )+n^{-2}{ϵ}^{-3}\lesssim \sum _{n=1}^{\infty }\frac{e^{-n{ϵ}^2/2{\sigma }^2}}{\sqrt{n}ϵ/\sigma }+n^{-2}{ϵ}^{-3}<\infty ,$$

where we use Mill’s Ratio. Thus, by the Borel-Cantelli Lemma, we have $P({\mathrm{limsup}}_{n\to \infty }|{\overline{X}}_n-\mu |>ϵ)=0$ and thus ${\overline{X}}_n\stackrel{\text{a.s.}}{\to }\mu$.

Footnotes

1. Michel, R. On the constant in the nonuniform version of the Berry-Esséen theorem. Z. Wahrscheinlichkeitstheorie verw Gebiete 55, 109–117 (1981). ↩

2. $c\le 33/4+8(1+e)$ ↩