Method of Moments

Suppose that we have $m$ unknown parameters to be estimated: $\theta =({\theta }_1,\dots ,{\theta }_m)$. Then we can use the first $m$ moments of the distribution and then equate them to the observations (empirical moments):

$$\{\begin{array}{ll}{\mu }_1({\theta }_1,\dots ,{\theta }_m)& ={\hat{\mu }}_1,\\ & ⋮\\ {\mu }_m({\theta }_1,\dots ,{\theta }_m)& ={\hat{\mu }}_m.\end{array}$$

This gives us a system of $m$ equations with $m$ unknowns, which can be solved to obtain the estimates ${\hat{\theta }}_1,\dots ,{\hat{\theta }}_m$. If we denote the RHS of the system compactly as $M(\theta )=({\mu }_1,\dots ,{\mu }_m)$ and assume $M$ is one to one, then ${\theta }^{(\mathrm{MM})}=M^{-1}({\hat{\mu }}_1,\dots ,{\hat{\mu }}_m)$.

• 📗 The simplest example is that ${\mu }_1={\theta }_1$, and then ${\hat{\theta }}_1=\bar{x}$. For variance, we have ${\hat{\theta }}_2=\sqrt{\overline{x^2}-{\overline{x}}^2}$.

Asymptotic Normality

Under the following regularity conditions:

• $M^{-1}$ is continuously differentiable at $M(\theta )$.
• The covariance matrix $\Sigma (\theta )={\mathrm{Cov}}_{\theta }(X_1,X_1^2,\dots ,X_1^m)$ exists.

Then the multivariate CLT and Delta Method yields

$$\sqrt{n}\left({\theta }_n^{(\mathrm{MM})}-\theta \right)\stackrel{d}{\to }\mathcal{N}(0,\Gamma (\theta )),$$

where

$$\Gamma (\theta )={\left(\frac{\partial M^{-1}}{\partial \theta }(M(\theta ))\right)}^T\Sigma (\theta )\left(\frac{\partial M^{-1}}{\partial \theta }(M(\theta ))\right).$$

General MM Estimator

Using moments is just one convenient way to construct linearly independent equations. One can choose other functions $g_1,\dots ,g_m:\mathcal{X}\to \mathbb{R}$, giving

$$\{\begin{array}{ll}{\mathbb{E}}_{\theta }[g_1(X)]& ={\hat{\mathbb{E}}}_n[g_1(X)]=\frac{1}{n}{\sum }_{i=1}^n{g}_1(X_i),\\ & ⋮\\ {\mathbb{E}}_{\theta }[g_m(X)]& ={\hat{\mathbb{E}}}_n[g_m(X)]=\frac{1}{n}{\sum }_{i=1}^n{g}_m(X_i).\end{array}$$

Write $g=(g_1,\dots ,g_m)$ as a vector-valued function, then ${\theta }^{(\mathrm{MM})}$ solves

$${\mathbb{E}}_{\theta }g={\hat{\mathbb{E}}}_{n}g$$

Similar results in Asymptotic Normality apply.

Misspecified Model

Asymptotic Normality also holds for misspecified models, i.e., when the true distribution is not in the model family. Suppose $M:\theta \mapsto {\mathbb{E}}_{\theta }g$ is well-defined, and $M^{-1}$ is continuously differentiable at ${\mathbb{E}}_{P}g$. Note that we do not need $P\in \{P_{\theta }{\}}_{\theta \in \Theta }$. Additionally, suppose $\Sigma ={\mathrm{Cov}}_P(g)$ exists. Then we have

$$\sqrt{n}\left({\theta }_n^{(\mathrm{MM})}-{\theta }^{\ast }\right)\stackrel{d}{\to }\mathcal{N}(0,\Gamma ),$$

where

$${\theta }^{\ast }=M^{-1}({\mathbb{E}}_{P}g),\Gamma =(M^{-1}{)}^{\prime }({\mathbb{E}}_{P}g{)}^T\Sigma (M^{-1}{)}^{\prime }({\mathbb{E}}_{P}g).$$