Regularity Conditions for M-Estimators

Regularity Conditions in Estimation

Common in Statistics, many core ideas in certain analyses are intuitive and simple. For example, the proofs of the Delta Method and asymptotic normality of M-estimators primarily rely on Taylor expansions. However, to make the proof rigorous, numerous “regularity conditions” are needed. These conditions have several characteristics:

• Not the main focus of the analysis
• Make functions or variables behave well, thereby validating the proof
• Most examples of interest satisfy these conditions
• Not tight: some conditions may be relaxed, but doing so may complicate the statement or proof
• Not unique: there can be different sets of regularity conditions that work for the same proof
• Different sets of regularity conditions may yield results with varying levels of generality or applicability

Some general regularity conditions used in estimation include:

1. Uniform support: The support of the PDF $f(x;\theta )$ does not depend on $\theta$.
2. Identifiability: $f(x;{\theta }_1)\ne f(x;{\theta }_2)⟺{\theta }_1\ne {\theta }_2$.
3. Interior: The parameter space $\Theta$ is finite or an open interval; the true parameter ${\theta }^{\ast }$ is not on the boundary of $\Theta$.

For Maximum Likelihood Estimation

• 💡 MLE is a special M-Estimator. One can also review the regularity conditions for M-Estimators. Here we discuss the regularity conditions for MLE specifically.

MLE considers the Log-Likelihood ${\ell }_{\theta }:=\log f(x;\theta )$ as the objective function.

Consistency

Similar to M-Estimators, we need a stronger identifiability condition and a stronger LLN condition:

1. Separation: For any $ϵ>0$, ${\sup }_{\theta :\Vert \theta -{\theta }^{\ast }\Vert >ϵ}\mathbb{E}{\ell }_{\theta }<\mathbb{E}{\ell }_{{\theta }^{\ast }}$.
2. Uniform convergence: ${\sup }_{\theta }|{\hat{\mathbb{E}}}_n{\ell }_{\theta }-\mathbb{E}{\ell }_{\theta }|\stackrel{P}{\to }0$.

• ❗️ Throughout the note, we do not need well-specification. The expectation is taken under the data generating distribution $p_{\mathrm{data}}$, and ${\theta }^{\ast }={\mathrm{argmin}}_{\theta \in \Theta }{\mathbb{E}}_{p_{\mathrm{data}}}{\ell }_{\theta }$.

Under (separation) and (uniform convergence), the MLE is consistent: ${\hat{\theta }}_{\mathrm{MLE}}\stackrel{P}{\to }{\theta }^{\ast }$.

---

We list two sufficient conditions

1. Compactness: $\Theta$ is compact.
2. Continuity: ${\ell }_{\theta }$ and $\mathbb{E}{\ell }_{\theta }$ are continuous in $\theta$.
3. Lower bound: $\mathbb{E}[{\inf }_{\theta \in \Theta }{\ell }_{\theta }(X)]>-\infty$

• Under (compactness), (identifiability) is equivalent to (separation).
• Under (compactness), (continuity) and (lower bound) implies (uniform convergence).

Asymptotic Normality

For asymptotic normality, we aim for the stronger result:

$$\sqrt{n}\left({\hat{\theta }}_{\mathrm{MLE}}-{\theta }_{\ast }\right)\stackrel{d}{\to }\mathcal{N}(0,I({\theta }_{\ast }{)}^{-1}),$$

where $I(\theta )$ is the Fisher Information matrix at . For this stronger result to hold, we generally requires strong continuity conditions that allow the exchange of integration and differentiation, useful in the calculation of the Fisher Information.

Beyond (identifiability), (uniform support), (interior), and the Consistency property, we need:

1. Smoothness: $f(x;\theta )$ is a thrice differentiable w.r.t $\theta$, and $|{\partial }_{\theta }^{3}f(x;\theta )|\le h(x)$ on some neighborhood of ${\theta }^{\ast }$ for some integrable function $h(x)$.
2. Exchangeability: We can interchange differentiation w.r.t $\theta$ up to second order and integration over $x$.
3. Invertibility: The Fisher Information matrix is invertible in a neighborhood of ${\theta }^{\ast }$.

The above set of regularity conditions are sufficient but not necessary. For example, the (smoothness) condition can be relaxed to first-order continuous differentiability with some additional conditions on the mapping $\theta \mapsto \mathbb{E}{\ell }_{\theta }$. And (exchangeability) can be implied by (smoothness) and some additional conditions. See M-Estimator for a more general set of regularity conditions.