Exponential Family

A family of univariate PDF/PMF is said to be exponential if it can be expressed as:

$$f(x;\theta )=c(\theta )h(x)e^{t(x)q(\theta )}$$

where $c(\theta )\ge 0,h(x)\ge 0$. A joint PDF for a multivariate distribution is called exponential if

$$f(x;\theta )=c(\theta )h(\mathbf{x})e^{{\sum }_{j=1}^k{t}_j(\mathbf{x})q_j(\theta )}$$

If $\theta \in {\mathbb{R}}^k$, such distribution is said to be in a $k$-parameter exponential family.

• $c(\theta )$ is the normalizing constant.
• $h(x)$ is the base measure.
• $\exp (q(\theta {)}^{T}t(x))$ is the exponential tilt that up(down)-weights the base measure.
• $t(x)$ is a sufficient statistic w.r.t. the natural parameter space

$$\Theta =\left\{\theta :\int h(x)e^{q(\theta {)}^{T}t(x)}\mathrm{d}x<\infty \right\}.$$

To verify if some family of distributions is of exponential type, we must be able to identify the functions $c(\theta ),h(x),t(x)$ and $q(\theta )$.

Examples:

• Bernoulli Distribution: $f(x;p)=(1-p)e^{x\log \frac{p}{1-p}}$
• Binomial Distribution: $f(x;n,p)=\left(\genfrac{}{}{0ex}{}{n}{x}\right)p^x(1-p{)}^{n-x}$
• Poisson Distribution: $p(n;\lambda )=e^{-\lambda }(n!{)}^{-1}{e}^{n\log \lambda }$
• Normal Distribution: $(2\pi {)}^{-n/2}{\sigma }^{-n}{e}^{n{\mu }^2}{e}^{-\left(\sum x_i^2-2\left(\sum x_i\right)\mu \right)/{\sigma }^2}$
• Exponential Distribution

MLE for Exponential Family

Due to the exponential family’s special form of the likelihood, the MLE estimator of $\theta$ coincides with the moment estimator w.r.t. the exponent statistic $t(x)$. WLOG, suppose $q(\theta )=\theta \in {\mathbb{R}}^k$. We first note that the inverse normalizing constant is infinitely differentiable:

$$\frac{{\partial }^p}{{\partial }^{j_1}{\theta }_1\dots {\partial }^{j_k}{\theta }_k}\left(\frac{1}{c(\theta )}\right)=\int h(x)t_1^{j_1}(x)\dots t_k^{j_k}(x)e^{{\theta }^{T}t(x)}\mathrm{d}x,$$

where $j_i\in \mathbb{N}$ and $p=\sum j_i$. Therefore, the derivative of the log-likelihood $\mathrm{LL}(\theta \mid x)$ is

$$\begin{array}{rl}{\mathrm{LL}}^{\prime }(\theta )=& \frac{\mathrm{d}}{\mathrm{d}\theta }\left(\log \left(c(\theta )h(x)e^{{\theta }^{T}t(x)}\right)\right)\\ =& \frac{c^{\prime }(\theta )}{c(\theta )}+t(x)\\ =& -c(\theta )\frac{\mathrm{d}}{\mathrm{d}\theta }\left(\frac{1}{c(\theta )}\right)+t(x)\\ =& t(x)-\int c(\theta )h(x)t(x)e^{{\theta }^{T}t(x)}\mathrm{d}x\\ =& t(x)-{\mathbb{E}}_{\theta }[t(X)].\end{array}$$

Then, the MLE estimator of $t$, which is the zero of the derivative, satisfies ${\mathbb{E}}_{{\hat{\theta }}_{\mathrm{MLE}}}t(X)={\hat{\mathbb{E}}}_{n}t(x)$, indicating that MLE is a General MM Estimator w.r.t $t(x)$. As a result, the asymptotic normality property of both MLE (M-estimator) and MM (Z-estimator) applies.