Exponential Family

A family of univariate PDF/PMF is said to be exponential if it can be expressed as:

$$f(x;\theta )=c(\theta )h(x)\exp (t(x)q(\theta ))$$

or equivalently,

$$f(x;\theta )=h(x)\exp (t(x)q(\theta )-b(\theta )),$$

where all values are scalars, $h(x)\ge 0$, and $c(\theta )=e^{-b(\theta )}\ge 0$ is the normalizing constant.

We can extend this to cover multivariate random variables and multi-dimensional parameters. Specifically, consider real vectors $x\in {\mathbb{R}}^d$, $\theta \in {\mathbb{R}}^s$ and $t,q\in {\mathbb{R}}^k$ where $k\ge s$. A joint PDF/PMF is said to be exponential if

$$f(x;\theta )=c(\theta )h(x)\exp (⟨t(x),q(\theta )⟩),$$

where the inner product can be matrix inner product. Common distributions have $s=k$, and such distribution is said to be in a ==$k$-parameter exponential family==. When $s<k$, we say it is in a curved exponential family. An exponential family has the following components:

• $c(\theta )$ is the normalizing constant;
• $h(x)$ is the base measure;
• $q(\theta )$ is the natural parameter; it can be thought of as a reparameterization of $\theta$, and thus we require the dimension of to be no less than that of $\theta$;
• $t(x)$ is a ==Sufficient Statistic== w.r.t. the natural parameter space:

$$\Theta =\left\{\theta :\int h(x)e^{⟨q(\theta ),t(x)⟩}\mathrm{d}x<\infty \right\};$$

• $\exp (q(\theta {)}^{T}t(x))$ is the exponential tilt that up(down)-weights the base measure.

If $q=\theta$ (perhaps after some reparameterization $\theta \leftarrow q(\theta )$), we say the exponential family is in canonical form. Further, if the sufficient statistic is the r.v. itself, i.e., $t(x)=x$, we say the exponential family is in natural form. In between, we have the dispersion form:

$$f(x;\theta )=h(\varphi ,x)\exp \left(\frac{x^T\theta -b(\theta )}{\varphi }\right),$$

where $\varphi$ is called the dispersion parameter. We can see that if $\varphi$ is known, then $\theta$ is the only canonical parameter, but the sufficient statistic is the dispersed $x$: $t(x)=x/\varphi$. If $\varphi$ is unknown, the model may corresponds to a multi-parameter exponential family.

• 📗 For example, a Normal Distribution with known variance ${\sigma }^2$ has a dispersion form with $\varphi ={\sigma }^2$, $t(x)=x$, $q(\mu )=\mu$, and $b(\mu )={\mu }^2/2$.

Examples

To verify if some family of distributions is of exponential type, we must be able to identify the functions $c(\theta )$ (or $b(\theta )$), $h(x)$, $t(x)$ and $q(\theta )$.

Distribution \ Component	PDF/PMF	$q$	$t$	$b$	$h$
Univariate Normal Distribution	$(2\pi {)}^{-1/2}{\sigma }^{-1}\exp \left(-(x-\mu {)}^2/(2{\sigma }^2\right)$	$(\mu /{\sigma }^2,-1/(2{\sigma }^2))$	$(x,x^2)$	${\mu }^2/(2{\sigma }^2)+\ln \sigma$	$(2\pi {)}^{-1/2}$
Multivariate Normal Distribution	$(2\pi {)}^{-d/2}|\Sigma {|}^{-1/2}\exp (-(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )/2)$	$({\Sigma }^{-1}\mu ,-{\Sigma }^{-1}/2)\in {\mathbb{R}}^d\times {\mathbb{R}}^{d\times d}$	$(x,x{x}^T)$	$({\mu }^T{\Sigma }^{-1}\mu +\ln |\Sigma |)/2$	$(2\pi {)}^{-d/2}$
Exponential Distribution	$\lambda \exp (-\lambda x)$	$-\lambda$	$x$	$-\ln \lambda$	${\mathrm{𝟙}}_{x\ge 0}$
Bernoulli Distribution	$p^x(1-p{)}^{1-x}$	$\ln (p/(1-p))$	$x$	$-\ln (1/(1-p))$	${\mathrm{𝟙}}_{x\in \{0,1\}}$
Binomial Distribution with known $n$	$\left(\genfrac{}{}{0ex}{}{n}{x}\right)p^x(1-p{)}^{n-x}$	$\ln (p/(1-p))$	$x$	$-n\ln (1/(1-p))$	$\left(\genfrac{}{}{0ex}{}{n}{x}\right){\mathrm{𝟙}}_{x\in \{0,\dots ,n\}}$
Poisson Distribution	${\lambda }^x{e}^{-\lambda }/x!$	$\log \lambda$	$x$	$\lambda$	$1/x!{\mathrm{𝟙}}_{x\in \mathbb{N}}$
Chi-Square Distribution	$e^{-x/2}(x/2{)}^{\nu /2-1}/(2\Gamma (\nu /2))$	$\nu /2-1$	$\ln x$	$\ln \Gamma (\nu /2)+\ln 2\cdot \nu /2$	$e^{-x/2}(\nu /2){\mathrm{𝟙}}_{x>0}$
Gamma Distribution	$\beta e^{-\beta x}(\beta x{)}^{\alpha -1}/\Gamma (\alpha )$	$(\alpha -1,-\beta )$	$(\ln x,x)$	$\ln \Gamma (\alpha )-\alpha \ln \beta$	${\mathrm{𝟙}}_{x>0}$
Beta Distribution	$x^{\alpha -1}(1-x{)}^{\beta -1}\Gamma (\alpha +\beta )/(\Gamma (\alpha )\Gamma (\beta ))$	$(\alpha ,\beta )$	$(\ln x,\ln (1-x))$	$\ln (\Gamma (\alpha )\Gamma (\beta )/\Gamma (\alpha +\beta ))$	${\mathrm{𝟙}}_{x\in (0,1)}/(x(1-x))$
Categorical Distribution with known $K$	${\prod }_{i=1}^K{p}_i^{x_i}$ with $p_K=1-{\sum }_{i=1}^{K-1}{p}_i$	$(\log (p_1/p_K),\dots ,\log (p_{K-1}/p_K))\in {\mathbb{R}}^{K-1}$	$(x_1,\dots ,x_{K-1})$	$-\ln p_K$	${\mathrm{𝟙}}_{x\in \{e_1,\dots ,e_K\}}$

MLE for Exponential Family

Due to the exponential family’s special form of the likelihood, the MLE estimator of $\theta$ coincides with the moment estimator w.r.t. the exponent statistic $t(x)$. WLOG, suppose $q(\theta )=\theta \in {\mathbb{R}}^k$. We first note that the inverse normalizing constant is infinitely differentiable:

$$\frac{{\partial }^p}{{\partial }^{j_1}{\theta }_1\dots {\partial }^{j_k}{\theta }_k}\left(\frac{1}{c(\theta )}\right)=\int h(x)t_1^{j_1}(x)\dots t_k^{j_k}(x)e^{{\theta }^{T}t(x)}\mathrm{d}x,$$

where $j_i\in \mathbb{N}$ and $p=\sum j_i$. Therefore, the derivative of the log-likelihood $\mathrm{LL}(\theta \mid x)$ is

$$\begin{array}{rl}{\mathrm{LL}}^{\prime }(\theta )=& \frac{\mathrm{d}}{\mathrm{d}\theta }\left(\log \left(c(\theta )h(x)e^{{\theta }^{T}t(x)}\right)\right)\\ =& \frac{c^{\prime }(\theta )}{c(\theta )}+t(x)\\ =& -c(\theta )\frac{\mathrm{d}}{\mathrm{d}\theta }\left(\frac{1}{c(\theta )}\right)+t(x)\\ =& t(x)-\int c(\theta )h(x)t(x)e^{{\theta }^{T}t(x)}\mathrm{d}x\\ =& t(x)-{\mathbb{E}}_{\theta }[t(X)].\end{array}$$

Then, the MLE estimator of $t$, which is the zero of the derivative, satisfies ${\mathbb{E}}_{{\hat{\theta }}_{\mathrm{MLE}}}t(X)={\hat{\mathbb{E}}}_{n}t(x)$, indicating that MLE is a General MM Estimator w.r.t $t(x)$. As a result, the asymptotic normality property of both MLE (M-estimator) and MM (Z-estimator) applies.

Moments of Dispersion Exponential Family

The above calculation also gives a convenient way to compute the first (mean) and second (variance) central moments of $X$ when $X$ follows a dispersion exponential family. Specifically, notice that $t(x)=x/\varphi$ and $c(\theta )=\exp (-b(\theta )/\varphi )$. Therefore, we have

$${\mathrm{LL}}^{\prime }(\theta )=\frac{c^{\prime }(\theta )}{c(\theta )}+t(x)=\frac{x-b^{\prime }(\theta )}{\varphi }.$$

Under For Maximum Likelihood Estimation, we have $\mathbb{E}[{\mathrm{LL}}^{\prime }(\theta )]=0$, and thus

$$\mathbb{E}[X]=b^{\prime }(\theta ).$$

Similarly, under the same conditions, we have

$$0=\mathbb{E}\left[{\mathrm{LL}}^{\prime \prime }(\theta )+({\mathrm{LL}}^{\prime }(\theta ){)}^2\right]=\frac{-b^{\prime \prime }(\theta )}{\varphi }+\frac{\mathbb{E}[(X-\mathbb{E}[X]{)}^2]}{{\varphi }^2},$$

which gives

$$\mathrm{Var}(X)=\varphi b^{\prime \prime }(\theta ).$$

• 📗 For example, for a Poisson Distribution, $\theta =\ln \lambda$, $b(\theta )=e^{\theta }$, and $\varphi =1$. Thus, $\mathbb{E}[X]=\mathrm{Var}(X)=b(\theta )=b^{\prime }(\theta )=b^{\prime \prime }(\theta )=\lambda$.