Z-Estimator

Recall that an M-Estimator seeks the minimizer of a function $M_n$. Suppose the function is differentiable and convex, its minimizer equals the zero of its derivative

$${\Psi }_n(\hat{\theta })=\frac{1}{n}\sum _{i=1}^n{\psi }_{\hat{\theta }}(X_i)=0,\text{\hspace{1em}(Z)}$$

where in this case, $\psi$ is the derivative of $m$ in the definition of M-estimator. For example, MLE is an estimator w.r.t the Score Function ${\psi }_{\theta }=\frac{\partial }{\partial \theta }\log p_{\theta }(X)$.

However, $\psi$ can be more general without necessarily corresponding to an optimization problem. For example, recall that for a Moment Estimator, we solve a system

$$\frac{1}{n}\sum _{i=1}^n{\psi }_{\hat{\theta }}(X_i)=\frac{1}{n}\sum _{i=1}^n\left({\mathbb{E}}_{\hat{\theta }}g-g(X_i)\right)={\mathbb{E}}_{\hat{\theta }}g-{\hat{\mathbb{E}}}_{n}g=0.$$

Thus, Moment Estimators are also a special case of Z-estimators.

As we can see, Z-estimators are a class of more general estimators that solve the zero point of a system of estimating equations $(Z)$.

• Table. Comparison of optimization and system solution.

Optimization	System Solution
M-estimators	Z-estimators
Optimizing an objective function	Solving an equation system
Utilize optimization landscape, e.g., gradient	Utilize system dynamics, e.g., contraction

As we discussed earlier, for convex/concave and differentiable objective functions, optimization is equivalent to solving a system regarding the gradient. Conversely, we can also define an objective function for solving a system of equations. For example, for a linear system $Ax=b$, we can define the squared cost $f(x)=\Vert Ax-b{\Vert }_2^2$, whose minimizer is the solution of the system.

Equivalence of optimization and system solution.

However, different problem formulations offer different insights and solution methods.

• Optimization is more suitable if you have a clear and well-motivated objective function;
• System solution is more suitable when you know how the solution determines the system dynamics.
• When optimizing a function, we usually care more about how the local landscape, e.g., gradient, carries the decision variable to the optimum;
• When solving a system, we usually want to follow some system dynamics, e.g., a contractive operator, to reach the solution.

Properties

Asymptotic Normality

Let ${\theta }^{\ast }$ solve the system $\mathbb{E}[{\psi }_{{\theta }^{\ast }}(X)]=0$. Suppose the consistency holds: $\hat{\theta }\stackrel{P}{\to }{\theta }^{\ast }$. Then, under some regularity conditions:

• $\theta \mapsto {\psi }_{\theta }(x)$ is twice differentiable for all $\theta$ and $\Vert {\ddot{\psi }}_{\theta }(x)\Vert \le f(x)$ for some integrable function $f$;¹
  • or, there exists an $L_2$ function $g$ such that for any ${\theta }_1,{\theta }_2$ in a neighborhood of ${\theta }^{\ast }$, we have $\Vert {\psi }_{{\theta }_1}(x)-{\psi }_{{\theta }_2}(x)\Vert \le g(x)\Vert {\theta }_1-{\theta }_2\Vert$;
• $\mathbb{E}{\dot{\psi }}_{\theta }(X)$ exists and is non-singular in a neighborhood of ${\theta }^{\ast }$;
• $\mathbb{E}{\psi }_{{\theta }^{\ast }}{\psi }_{{\theta }^{\ast }}^T$ exists;

we have

$$\sqrt{n}\left(\hat{\theta }-{\theta }^{\ast }\right)\stackrel{d}{\to }\mathcal{N}(0,V_{{\theta }^{\ast }}^{-1}\mathbb{E}[{\psi }_{{\theta }^{\ast }}{\psi }_{{\theta }^{\ast }}^T]V_{{\theta }^{\ast }}^{-1}),$$

where $V_{{\theta }^{\ast }}=\frac{\partial }{\partial \theta }\mathbb{E}{\psi }_{\theta }(X){|}_{{\theta }^{\ast }}$.

Relation to Asymptotic Normality of M-Estimators

If ${\psi }_{\theta }={\dot{m}}_{\theta }$, where $m$ is the objective function for an M-Estimator, we can see that the asymptotic normality of Z-estimators implies the asymptotic normality of M-estimators.

However, the regularity conditions for Z-estimators are stronger than those for M-estimators (see Asymptotic Normality). For example, we can show that quantile regression satisfies the asymptotic normality conditions for M-estimators; but it does not satisfy the conditions for Z-estimators.

Proof Sketch

Denote $\Psi (\theta ):=\mathbb{E}{\psi }_{\theta }(X)$; recall that ${\Psi }_n(\theta ):={\hat{\mathbb{E}}}_n{\psi }_{\theta }(X)$. By Taylor expansion,

$$0={\Psi }_n(\hat{\theta })={\Psi }_n({\theta }^{\ast })+{\dot{\Psi }}_n({\theta }^{\ast })(\hat{\theta }-{\theta }^{\ast })+o_P(\Vert \hat{\theta }-{\theta }^{\ast }\Vert ).$$

Since we assume the consistency, we get

$$\sqrt{n}(\hat{\theta }-{\theta }^{\ast })\stackrel{P}{\to }-{\dot{\Psi }}_n({\theta }^{\ast }{)}^{-1}(\sqrt{n}{\Psi }_n({\theta }^{\ast })).$$

By LLN, ${\dot{\psi }}_n({\theta }^{\ast }{)}^{-1}\to \dot{\psi }({\theta }^{\ast }{)}^{-1}$; by CLT,

$$\sqrt{n}{\Psi }_n({\theta }^{\ast })\stackrel{d}{\to }\mathcal{N}(0,\mathbb{E}[{\psi }_{{\theta }^{\ast }}{\psi }_{{\theta }^{\ast }}^T]).$$

Thus, by Slutsky’s theorem,

$$\sqrt{n}(\hat{\theta }-{\theta }^{\ast })\stackrel{d}{\to }\mathcal{N}l(0,\dot{\Psi }({\theta }^{\ast }{)}^{-1}\mathbb{E}[{\psi }_{{\theta }^{\ast }}{\psi }_{{\theta }^{\ast }}^T]\dot{\Psi }({\theta }^{\ast }{)}^{-1}).$$

Asymptotic Normality of Least Squares

We cast Ordinary Least Squares as a Z-estimator and verify the asymptotic normality conditions. The cost function is $m_{\theta }(x,y)=(y-{\theta }^{T}x{)}^2$, which gives the z-function ${\psi }_{\theta }(x,y)=2x(y-x^T\theta )$.

For the first condition, we verify its alternative Lipschitz condition:

$$\Vert {\psi }_{{\theta }_1}(x,y)-{\psi }_{{\theta }_2}(x,y)\Vert \le 2\Vert x{x}^T\Vert \Vert {\theta }_1-{\theta }_2\Vert .$$

Suppose $X$ has finite moments, then the first condition is met.

For the second condition, we have

$$\mathbb{E}{\dot{\psi }}_{\theta }(X)=-2\mathbb{E}[X{X}^T].$$

For the third condition, we have

$$\mathbb{E}{\psi }_{{\theta }^{\ast }}{\psi }_{{\theta }^{\ast }}^T=\mathbb{E}\left[4(Y-X^T{\theta }^{\ast }{)}^{2}X{X}^T\right].$$

Therefore, by the asymptotic normality of Z-estimators, we have

$$\sqrt{n}(\hat{\theta }-{\theta }^{\ast })\stackrel{d}{\to }\mathcal{N}\left(0,(\mathbb{E}X{X}^T{)}^{-1}\mathbb{E}\left[(Y-X^T{\theta }^{\ast }{)}^{2}X{X}^T\right](\mathbb{E}X{X}^T{)}^{-1}\right).$$

Footnotes

1. The measure is always the probabilistic measure. ↩