Central Limit Theorem

Let $\{X_i{\}}_{i=1}^n$ be a set of i.i.d. Random Variables with same Mean $\mu$ and Variance ${\sigma }^2<+\infty$. Denote $\overline{X}={\sum }_{i=1}^n{X}_i/n$. Then

$$\frac{\overline{X}-\mu }{\sigma /\sqrt{n}}\stackrel{d}{⟶}Z\sim \mathcal{N}(0,1),$$

where $d$ means convergence in distribution.

• 💡 Also holds for multi-variate distributions: $\sqrt{n}(\overline{X}-\mu )\stackrel{d}{\to }\mathcal{N}(0,\Sigma )$.

Proof

If $X_i$ has MGF, we can use MGF to prove the theorem. To be more general, we use Characteristic Function. WLOG, we can assume $\mu =0,\sigma =1$. Let $\varphi (t)$ be the characteristic function of $X_i$. Then we have

$$\varphi (0)=1,{\varphi }^{\prime }(0)=i\mathbb{E}[X_i]=0,{\varphi }^{\prime \prime }(0)=-\mathbb{E}[X_i^2]=-1.$$

Therefore,

$$\underset{t\to 0}{\lim }\frac{\varphi (t)-\left(1-\frac{1}{2}{t}^2\right)}{t^2}=\underset{t\to \infty }{\lim }\frac{1-\frac{1}{2}{t}^2+o(t^2)-\left(1-\frac{1}{2}{t}^2\right)}{t^2}=0.$$

And the characteristic function of $\sqrt{n}\overline{X}$ is

$${\varphi }_{\overline{X}}(t)=\mathbb{E}[e^{it\sqrt{n}\overline{X}}]=\prod _{i=1}^n\mathbb{E}[e^{it/\sqrt{n}{X}_i}]=\varphi {\left(\frac{t}{\sqrt{n}}\right)}^n.$$

Then we have

$$\underset{n\to \infty }{\lim }\left|\varphi {\left(\frac{t}{\sqrt{n}}\right)}^n-{\left(1-\frac{t^2}{2n}\right)}^n\right|\le \underset{t\to \infty }{\lim }n\left|\varphi \left(\frac{t}{\sqrt{n}}\right)-\left(1-\frac{t^2}{2n}\right)\right|=\underset{n\to \infty }{\lim }{t}^2\cdot 0=0.$$

Therefore,

$$\underset{n\to \infty }{\lim }{\varphi }_{\overline{X}}(t)=\underset{n\to \infty }{\lim }{\left(1-\frac{t^2}{2n}\right)}^n=e^{-\frac{1}{2}{t}^2},$$

which is the CF of standard Normal Distribution. By the inverse property and Convergence of Characteristic Functions,

$$\sqrt{n}\overline{X}\stackrel{d}{⟶}Z\sim \mathcal{N}(0,1).$$

Sup-Norm Approximation Error: Berry-Essèen Theorem

Suppose $\mathbb{E}|X_1{|}^3<+\infty$. Then

$$\underset{t}{\sup }\left|P(Z_n\le t)-\Phi (t)\right|\le \frac{33}{4}\frac{\mathbb{E}|X_1-\mu {|}^3}{\sqrt{n}{\sigma }^3}.$$