Maximum Likelihood Estimation

Maximum likelihood estimation (MLE) provides a principled approach to estimating the parameters of a statistical model by maximizing the Likelihood function. It is general and has nice properties.

For a set of $m$ examples $X=\{x^i{\}}_{i=1}^m$ generated by data generating distribution $p_{\mathrm{data}}(x)$. And $p_{\mathrm{model}}(x;\theta )$ is a parametric family of model distributions estimating $p_{\mathrm{data}}$. We define the maximum likelihood estimator for $\theta$

$${\theta }_{\mathrm{ML}}:=\underset{\theta }{\mathrm{argmax}}{p}_{\mathrm{model}}(X;\theta )=\underset{\theta }{\mathrm{argmax}}\prod _{i=1}^m{p}_{\mathrm{model}}(x^i;\theta )$$

Rmk

• Here $p(x)=p(X_i=x)$ can be a PDF for PMF
• The latter equality requires the i.i.d. condition

Intuitively, since $X$ is generated by $p_{\mathrm{data}}$, $p_{\mathrm{data}}(x^i)$ should be relatively high. Then if $p_{\mathrm{model}}(x;\theta )$ is close enough to $p_{\mathrm{data}}$, $\prod p_{\mathrm{model}}(x^i;\theta )$ should be relatively high too. In the same spirit, $p_{\mathrm{model}}(x;\theta )$ is called the likelihood function, or written as

$$L_n:{\mathcal{X}}^n\times \Theta \to \mathbb{R},(x_1,\dots ,x_n;\theta )\mapsto p_{\theta }(X_1=x_1,\dots ,X_n=x_n).$$

Logarithm Trick

In practice, using the monotonically increasing property of the logarithm function, we often calculate the MLE by the following equivalences

$$\begin{array}{rl}{\theta }_{\mathrm{ML}}& =\underset{\theta }{\mathrm{argmax}}\prod _{i=1}^m{p}_{\mathrm{model}}(x^i;\theta )\\ & =\underset{\theta }{\mathrm{argmax}}\log \prod _{i=1}^m{p}_{\mathrm{model}}(x^i;\theta )\\ & =\underset{\theta }{\mathrm{argmax}}\sum _{i=1}^m\log p_{\mathrm{model}}(x^i;\theta )\\ & =\underset{\theta }{\mathrm{argmax}}{\mathbb{E}}_{x\sim {\hat{p}}_{\mathrm{data}}}\log p_{\mathrm{model}}(x;\theta )\end{array}$$

where ${\hat{p}}_{\mathrm{data}}$ is the Empirical Distribution defined by the training data. This is called the logarithm trick.

Relation With KL Divergence

Another interpretation of MLE is that it minimizes the KL Divergence, or Cross-Entropy, between $p_{\mathrm{data}}$ and $p_{\mathrm{model}}$, which measures the similarity between the two distributions:

$$\{\begin{array}{l}{D}_{\mathrm{KL}}({\hat{p}}_{\mathrm{data}}\Vert p_{\mathrm{model}})={\mathbb{E}}_{x\sim {\hat{p}}_{\mathrm{data}}}[\log {\hat{p}}_{\mathrm{data}}(x)-\log p_{\mathrm{model}}(x)]\\ H({\hat{p}}_{\mathrm{data}},p_{\mathrm{model}})=-{\mathbb{E}}_{x\sim {\hat{p}}_{\mathrm{data}}}[\log p_{\mathrm{model}}(x)]\end{array}$$

Note that ${\mathbb{E}}_{{\hat{p}}_{\mathrm{data}}}\to {\mathbb{E}}_{p_{\mathrm{data}}}$ by LLN.

Additionally, when doing the Logarithm Trick, we transform the product (joint distribution) into a sum (empirical distribution). This hints that KL Divergence tensorize (see ^rmk-tv-kl).

Conditional Log-Likelihood

The MLE can readily be generalized to the case where our goal is to estimate a conditional probability $P(y|x;\theta )$ in order to predict $y$ given $x$:

$${\theta }_{\mathrm{ML}}=\underset{\theta }{\mathrm{argmax}}P(Y|X;\theta ).$$

Here $X$ represents the inputs and $Y$ represents the observed targets. This forms the basis for most Supervised Learning methods, for example, Linear Regression.

If the samples are assumed to be i.i.d., then we have

$${\theta }_{\mathrm{ML}}=\underset{\theta }{\mathrm{argmax}}\sum _i\log P(y^i|x^i;\theta ).$$

Properties of MLE

Misspecification

Without special remark, the following properties hold for misspecified model, i.e., $p_{\mathrm{data}}\notin \{p_{\theta }{\}}_{\theta \in \Theta }$. From now on, we write ${\mathbb{E}}_p$ as the expectation under the data generating distribution $p_{\mathrm{data}}$.

Constancy/Invariance

For any function $g$, the transformation of a MLE $g(\hat{\theta })$ is still the MLE of $g(\theta )$.

Consistency

We denote ${\ell }_{\theta }=\log p_{\theta }$ as the log-likelihood, and let ${\theta }_{\ast }={\mathrm{argmax}}_{\theta \in \Theta }{\mathbb{E}}_p{\ell }_{\theta }(X)$.

Given sufficient regularity conditions, MLE is consistent: ${\hat{\theta }}_{\mathrm{ML}}\stackrel{P}{⟶}{\theta }_{\ast }$.

Further, if we have realizability and identifiability: $\exists !{\theta }_{\mathrm{data}}\in \Theta :p_{{\theta }_{\mathrm{data}}}=p_{\mathrm{data}}$, then ${\theta }_{\ast }={\theta }_{\mathrm{data}}$. This is because of the property of KL Divergence: $D_{\mathrm{KL}}(p_{\mathrm{data}}\Vert p_{\theta })=0$ if and only if $p_{\theta }=p_{\mathrm{data}}$.

Please refer to Consistency for the proof of a more general result.

Asymptotic Normality

Given sufficient regularity conditions, we have

$$\sqrt{n}\left({\hat{\theta }}_{\mathrm{ML}}-{\theta }_{\ast }\right)\stackrel{d}{\to }\mathcal{N}(0,{\Sigma }_{\mathrm{ML}})$$

where the asymptotic variance is

$${\Sigma }_{\mathrm{ML}}={\left({\mathbb{E}}_p{\ddot{\ell }}_{{\theta }_{\ast }}(X_i)\right)}^{-1}\left({\mathbb{E}}_p{\dot{\ell }}_{{\theta }_{\ast }}(X_i){\dot{\ell }}_{{\theta }_{\ast }}(X_i{)}^T\right){\left({\mathbb{E}}_p{\ddot{\ell }}_{{\theta }_{\ast }}(X_i)\right)}^{-T},$$

where the derivatives are taken with respect to $\theta$.

Suppose the model is well-specified, then the above variance can be simplified to

$${\hat{\theta }}_{\mathrm{ML}}-{\theta }_{\ast }\stackrel{d}{\to }\mathcal{N}(0,I_n^{-1}),$$

where $I_n$ is the Fisher Information.

• 📎 This property can be used to prove CLT when $\theta =\mu$, ${\hat{\theta }}_{\mathrm{ML}}=\overline{x}$ and $\sigma$ is known.

Please refer to Asymptotic Normality for the proof of a more general result.

Best Statistical Efficiency

We say a consistent estimator has better statistical efficiency, if it obtains lower generalization error for a fixed number of samples, or equivalently, requires fewer examples to obtain a fixed level of generalization error.

Formally, given two estimators

$$\{\begin{array}{l}\sqrt{n}({\hat{\theta }}_1-\theta )\stackrel{d}{\to }\mathcal{N}(0,{\Gamma }_1),\\ \sqrt{n}({\hat{\theta }}_2-\theta )\stackrel{d}{\to }\mathcal{N}(0,{\Gamma }_2),\end{array}$$

we say ${\hat{\theta }}_1$ is asymptotically more efficient than ${\hat{\theta }}_2$ if ${\Gamma }_1\prec {\Gamma }_2$.

The Cramér-Rao lower bound shows that no consistent estimator has a lower Mean Squared Error than the maximum likelihood estimator for a large number of samples.

To be more specific, we have the following two theorems.

Almost Everywhere Convolution Theorem

This theorem states that any consistent estimator converges to $Z_n+W_n$ where $Z_n\sim \mathcal{N}({\theta }_{\ast },I_n^{-1})$ and $Z_n\perp \perp W_n$, hence showing that MLE has the best efficiency.

Suppose the model $\mathcal{P}$ contains quadratic mean differentiable (QMD) distributions, and for all ${\theta }_{\ast }\in \Theta$¹, the estimator $\hat{\theta }$ satisfies

$$\sqrt{n}(\hat{\theta }-{\theta }_{\ast })\stackrel{d}{\to }{\mathrm{Law}}_{{\theta }_{\ast }}$$

Then, for almost all ${\theta }_{\ast }\in \Theta$, there exists some distribution $D_{{\theta }_{\ast }}$ such that

$${\mathrm{Law}}_{{\theta }_{\ast }}=\mathcal{N}(0,I_{{\theta }_{\ast }}^{-1})\ast D_{{\theta }_{\ast }}.$$

Alternatively, we can write

$$\hat{\theta }\stackrel{d}{\to }\sqrt{\frac{I_{{\theta }_{\ast }}^{-1}}{n}}(Z-{\theta }_{\ast })+\frac{W}{\sqrt{n}}={\theta }_{\mathrm{ML}}+\frac{W}{\sqrt{n}},$$

where $Z\sim \mathcal{N}(0,I),W\sim D_{{\theta }_{\ast }}$, and ${\theta }_{\mathrm{ML}}$ is the convergence point of MLE. Therefore, even if $W$ introduces zero bias, the additional variance introduced by it makes the estimator less efficient than MLE.

Local Asymptotic Minimax Theorem

Suppose the model is QMD, and the loss function $L:{\mathbb{R}}^d\to {\mathbb{R}}_{+}$ is bowl-shaped. Then, for any estimator,

$$\underset{\text{finite }I\subset {\mathbb{R}}^d}{\sup }\underset{n\to \infty }{\mathrm{liminf}}\underset{h\in I}{\sup }{\mathbb{E}}_{{\theta }_{\ast }+\frac{h}{\sqrt{n}}}L\left(\sqrt{n}\left(\hat{\theta }-{\theta }_{\ast }-\frac{h}{\sqrt{n}}\right)\right)\ge \mathbb{E}[L(Z)],Z\sim \mathcal{N}(0,I_{{\theta }_{\ast }}^{-1}).$$

The first three limiting operations correspond to “local”, “asymptotic”, and “minimax” respectively; they together characterize a sufficiently large neighborhood around ${\theta }_{\ast }$. Again, it states that the minimum risk achieved by MLE cannot be improved.

Footnotes

1. That is, the estimator is consistent regardless the location of ${\theta }_{\ast }$. For well-specified model, this is equivalent to that $\hat{\theta }$ is consistent for all data distributions $p_{{\theta }_{\ast }}$. ↩