Moment Generating Function

The moment generating function of a Random Variable is

$$M(t)=\mathbb{E}[e^{tX}],$$

where $\mathbb{E}$ is the Expectation. We note that

$$\frac{{\text{d}}^n}{\text{d}{t}^n}\mathbb{E}[e^{tX}]{|}_{t=0}=\mathbb{E}[X^n],$$

which is the $n$th moment of $X$.

• We only need the MGF to be defined around 0. Even though, there are some Random Variables that don’t have a MGF.
  • Cauchy Distribution doesn’t have a MGF except at $t=0$.

Inversion Theorem

Suppose that $M_X(s)$ is finite for all $s$ in an interval of the form $[-a,a]$, where $a>0$. Then, $M_X$ determines uniquely the CDF of the Random Variable $X$. In particular, if $M_X(s)=M_Y(s)<\infty$, for all $s\in [-a,a]$, then the random variables $X$ and $Y$ have the same CDF.

Multivariate Tranform

We can define joint MGF, or multivariate transform, of $n$ r.v.s $X_1,\dots ,X_n$:

$$M_{X_1,\dots ,X_n}(s_1,\dots ,s_n)=\mathbb{E}\left[\exp \left(\sum _{i=1}^n{s}_i{X}_i\right)\right].$$

The inversion theorem holds for the joint MGF as well: if $Y_1,\dots ,Y_n$ is another set of random variables and $M_{X_1,\dots ,X_n}\left(s_1,\dots ,s_n\right),M_{Y_1,\dots ,Y_n}\left(s_1,\dots ,s_n\right)$ are the same functions of $s_1,\dots ,s_n$, in a neighborhood of the origin, then the joint distribution of $X_1,\dots ,X_n$ is the same as the joint distribution of $Y_1,\dots ,Y_n$.

Properties

• $Y=aX+b⟹M_Y(s)=e^{sb}{M}_X(as)$
• $X\perp Y⟹M_{X+Y}(s)=M_X(s)M_Y(s)$
• $X\perp Y⟹M_{X,Y}(s,t)=M_X(s)M_Y(t)$
• $X\perp Y\wedge (Z=BX+(1-B)Y,B\sim \mathrm{Bern}(p))⟹M_X(s)=p{M}_X(s)+(1-p)M_Y(s)$

Example - MGF of a Normal Distribution

We calculate the MGF of a Normal Distribution. For a standard normal distribution $\mathcal{N}(0,1)$, one can easily calculate that $M_Z(s)=\exp (s^2/2)$. Using the first property, for a general normal distribution $X\sim \mathcal{N}(\mu ,{\sigma }^2)$, we have $M_X(s)=\exp (\mu s+{\sigma }^2{s}^2/2)$.

Further, for $X\sim \mathcal{N}({\mu }_1,{\sigma }_1^2)$ and $Y\sim \mathcal{N}({\mu }_2,{\sigma }_2^2)$, we have

$$M_{X+Y}(s)=\exp \left(({\mu }_1+{\mu }_2)s+({\sigma }_1^2+{\sigma }_2^2)s^2/2\right).$$

Note that this is the MGF of $\mathcal{N}({\mu }_1+{\mu }_2,{\sigma }_1^2+{\sigma }_2^2)$. By the inversion theorem, we conclude that $X+Y\sim \mathcal{N}({\mu }_1+{\mu }_2,{\sigma }_1^2+{\sigma }_2^2)$.

Related Problems

• Exercise 5 (Interchanging expectation and differentiation) justifies the interchange of expectation and differentiation in the computation of moments for nonnegative random variables.
• Exercise 3

Boundedness of MGF

Suppose

$$\underset{x\to \infty }{\mathrm{limsup}}\frac{\log \mathbb{P}(X>x)}{x}=-\nu <0.$$

Show that $M_X(s)$ is finite for all $s\in [0,\nu )$.

Solution:

First, the $\mathrm{limsup}$ implies that for any $ϵ>0$, there exists $x_0$ such that $\frac{\log \mathbb{P}(X>x)}{x}<-\nu +ϵ$for all $x>x_0$. We only need to show that $\mathbb{E}[e^{sX}\mid x>x_0]$ is finite. By the definition of Expectation for nonnegative random variables, we have

$$\mathbb{E}[e^{sX}\mid x>x_0]={\int }_{\exp s{x}_0}^{\infty }P(e^{sX}>t)\mathrm{d}t={\int }_{\exp s{x}_0}^{\infty }P(X>(\ln t)/s)\mathrm{d}t\le {\int }_{\exp s{x}_0}^{\infty }\exp ((ϵ-\nu )(\ln t)/s)\mathrm{d}t={\int }_{\exp s{x}_0}^{\infty }{t}^{(ϵ-\nu )/s}\mathrm{d}t.$$

Let $ϵ<\nu -s$. Then $(ϵ-\nu )/s<-1$, which gives

$${\int }_{\exp s{x}_0}^{\infty }{t}^{(ϵ-\nu )/s}\mathrm{d}t<\infty .$$