Delta Method

Given a converging sequence of r.v.s and a function, the delta method uses the function’s first-order derivative to describe the limiting distribution of the transformed sequence after the function.

Formally, suppose there exists a vector $\theta \in {\mathbb{R}}^k$ and a $k$-dimensional r.v. $T$, such that the number sequence $\{r_n\}$ and the r.v. sequence $\{{\theta }_n\}$ satisfy

$$r_n({\theta }_n-\theta )\stackrel{d}{\to }T,\text{and}{r}_n\to \infty ,\text{as}n\to \infty .$$

Additionally, suppose the function $\varphi :{\mathbb{R}}^k\to {\mathbb{R}}^d$ is differentiable at $\theta$. Then,

$$r_n\left(\varphi ({\theta }_n)-\varphi (\theta )\right)\stackrel{d}{\to }{\nabla }_{\theta }^T\varphi \cdot T.$$

Proof Sketch

We can approximate the difference $\varphi ({\theta }_n)-\varphi (\theta )$ using its first-order derivative:

$$\varphi ({\theta }_n)-\varphi (\theta )\approx {\nabla }_{\theta }^T\varphi ({\theta }_n-\theta ).$$

Thus,

$$r_n\left(\varphi ({\theta }_n)-\varphi (\theta )\right)\approx {\nabla }_{\theta }^T\varphi \left(r_n({\theta }_n-\theta )\right)\stackrel{d}{\to }{\nabla }_{\theta }^T\varphi \cdot T.$$

Formal Proof

Define the remainder $R(h)=\varphi (\theta +h)-\varphi (\theta )-{\nabla }_{\theta }^T\varphi h=o(\Vert h\Vert )$. Since $r_n\to \infty$, we know ${\theta }_n-\theta \stackrel{d}{\to }T/\infty =0$. By Slutsky’s Theorem, using the Stochastic Asymptotic Notation, we have

$$R({\theta }_n-\theta )=o_P(\Vert {\theta }_n-\theta \Vert ).$$

On the other hand, we have $r_n({\theta }_n-\theta )=O_P(1)$. Thus

$$r_{n}R({\theta }_n-\theta )=r_n\cdot o_P\left(\Vert {\theta }_n-\theta \Vert \right)=o_P\left(\Vert r_n({\theta }_n-\theta )\Vert \right)=o_P(O_P(1))=o_P(1).$$

Therefore

$$r_n\left(\varphi ({\theta }_n)-\varphi (\theta )\right)=r_n\left(R({\theta }_n-\theta )+{\nabla }_{\theta }^T\varphi ({\theta }_n-\theta )\right)\stackrel{d}{\to }{r}_n{\nabla }_{\theta }^T\varphi ({\theta }_n-\theta )\stackrel{d}{\to }{\nabla }_{\theta }^T\varphi \cdot T.$$

Asymptotic Normality

The delta method is often combined with CLT to establish asymptotic normality. Suppose $\sqrt{n}({\theta }_n-\theta )\stackrel{d}{\to }\mathcal{N}(0,\Sigma )$, then $\sqrt{n}(\varphi ({\theta }_n)-\varphi (\theta ))\stackrel{d}{\to }\mathcal{N}(0,{\varphi }^{\prime }(\theta )\Sigma {\varphi }^{\prime }(\theta {)}^T)$, where ${\varphi }^{\prime }$ is the Jacobian matrix.

The Jacobian matrix is the transpose of the gradient vector ${\nabla }_{\theta }\varphi$. That is, ${\varphi }^{\prime }(\theta )={\nabla }_{\theta }^T\varphi$.

Application

1. Write statistics as a (differentiable) function of simple statistics.
2. Apply CLT on simple statistics.
3. Apply the delta method on the function, to obtain the limiting distribution of the complex statistic.

Example - Sample Variance

Let the sample variance be ${\hat{\sigma }}^2=\frac{1}{n}\sum (x_i-\overline{x}{)}^2$. Suppose the sample has finite $k$th moment ${\alpha }_k$ up to $k\ge 4$. Then,

$$\sqrt{n}({\hat{\sigma }}^2-{\sigma }^2)\stackrel{d}{\to }\mathcal{N}\left(0,(-2{\alpha }_1,1)\left(\begin{array}{cc}{\alpha }_2-{\alpha }_1^2& {\alpha }_3-{\alpha }_1{\alpha }_2\\ {\alpha }_3-{\alpha }_1{\alpha }_2& {\alpha }_4-{\alpha }_2^2\end{array}\right)\left(\begin{array}{c}-2{\alpha }_1\\ 1\end{array}\right)\right).$$

To see this, we first express the sample variance as a function of the sample mean and the sample second moment:

$${\hat{\sigma }}^2=\varphi (\overline{x},\overline{x^2}),\text{where }\varphi (x,y)=y-x^2.$$

We have ${\varphi }^{\prime }(\mu ,{\sigma }^2)=(-2{\alpha }_1,1)$. And by CLT, we have

$$\sqrt{n}\left(\left(\begin{array}{c}\overline{X}\\ \overline{X^2}\end{array}\right)-\left(\begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}\right)\right)\stackrel{d}{\to }\mathcal{N}\left(0,\left(\begin{array}{cc}{\alpha }_2-{\alpha }_1^2& {\alpha }_3-{\alpha }_1{\alpha }_2\\ {\alpha }_3-{\alpha }_1{\alpha }_2& {\alpha }_4-{\alpha }_2^2\end{array}\right)\right).$$

Combining CLT and delta method gives the result.