Normal Distribution

A Random Variable is said to be normally distributed if it has

• Notation
  • $\mathcal{N}(\mu ,{\sigma }^2)$
• Parameters
  • $\mu \in \mathbb{R},{\sigma }^2\in {\mathbb{R}}_{+}$
• PDF
  • $f(x)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{1}{2}{\left(\frac{x-\mu }{\sigma }\right)}^2}$
  • $(2\pi {)}^{-k/2}|\mathbf{\Sigma }{|}^{-1/2}\exp \left(-\frac{1}{2}(\mathbf{x}-\mathbf{\mu }{)}^{\top }{\mathbf{\Sigma }}^{-1}(\mathbf{x}-\mathbf{\mu })\right)$ for $k$-dimensional with PSD $\Sigma$
• Mean
  • $\mu$
• Variance
  • ${\sigma }^2$
• MGF
  • $e^{\mu t+{\sigma }^2{t}^2/2}$
• CF
  • $e^{it\mu -{\sigma }^2{t}^2/2}$

Because of the Central Limit Theorem, in practice, many random phenomena obey, at least approximately, a normal probability distribution.

Equivalent Definitions for Multivariate Normal Distribution

We can define a random vector to be (nondegenerate multivariate) normal if if has a PDF specified above.

1. Or, if it has the form:

$$X=DW+\mu ,$$

for any matrix $D$ and vector $\mu$, where $W$ is a random vector whose components are independent standard normal random variables $\mathcal{N}(0,1)$. ^7bb02c 2. Or, if for any real vector $a$, the random variable $a^{T}X$ is normal.

These two alternative definitions cover degenerate normal distribution, i.e., some components are constant/ the distribution is concentrated on a proper subspace of ${\mathbb{R}}^k$.

Equivalence

First Alternative

By change of variables, we can see that the definition via PDF implies the first alternative definition. We now show that if $D$ is not singular, $X=DW+\mu$ is a nondegenerate normal r.v. with a PDF of the form above. By the relationship of Derived Distribution, we have

$$f_X(x)=f_W(D^{-1}x)\cdot |D^{-1}|.$$

Plugging in the PDF of standard normal distribution, we get

$$f_X(x)=(2\pi {)}^{-n/2}\exp \left(-\frac{1}{2}(x-\mu {)}^T{D}^{-T}{D}^{-1}(x-\mu )\right)\cdot \frac{1}{|D|}=(2\pi {)}^{-n/2}|D{D}^T{|}^{-1/2}\exp \left(-\frac{1}{2}(x-\mu {)}^T(D{D}^T{)}^{-1}(x-\mu )\right),$$

and $D{D}^T$ is the covariance of $X$.

Second Alternative

The relationship from the first alternative definition to the second is direct. Now suppose $a^{T}X$ is normal for any vector $a$.

Wrong

We inspect the MGF of $X$:

$$M_X(s)=M_{s^{T}X}(1)=\exp (s^T\mathbb{E}[X]+s^T\mathrm{Var}(X)s/2),$$

where we use the fact that $s^{T}X$ is normal with mean $s^T\mathbb{E}X$ and variance $s^T\mathrm{Var}(X)s$. On the other hand, let $Y=\sqrt{\mathrm{Var}(X)}W+\mathbb{E}[X]$, where $W$ is a standard normal random vector. Then, $Y$ is multivariate normal and has the same MGF as $X$. By the inversion theorem of MGF, $X$ and $Y$ have the same distribution.

• ❗️ The above proof is wrong, as we want to show $X=Y$ but not only $X\stackrel{d}{=}Y$.

To rigorously prove the equivalence, we consider two cases.

Case I. $V=D^2=\mathrm{Cov}(X,X)\succ 0$. Thus, $D$ is invertible and we let $W=D^{-1}(X-\mu )$. We want to show that $W$ is standard normal, and thus $X=DW+\mu$ satisfies the first alternative definition. We first have $\mathbb{E}[W]=0$ and $\mathrm{Var}(W)=D^{-1}\mathrm{Var}(X)D^{-T}=D^{-1}{D}^2{D}^{-1}=I$. We then inspect the MGF (transform) of $W$:

$$\begin{array}{rlrl}{M}_W(s)=& \mathbb{E}[\exp (s^{T}W)]& & \\ =& \mathbb{E}[\exp (s^{T}W+{\mu }^{\prime })\exp (-{\mu }^{\prime })]& & ({\mu }^{\prime }=s^T{D}^{-1}\mu )\\ =& \exp (-{\mu }^{\prime })\mathbb{E}[\exp (s^{\prime T}X)]& & (s^{\prime }=D^{-1}s)\\ =& \exp (-{\mu }^{\prime })M_{s^{\prime T}X}(1)& & \\ =& \exp (-{\mu }^{\prime })\exp (s^{\prime T}\mu +s^{\prime T}V{s}^{\prime }/2)& & (s^{\prime }X\text{ is normal by Alt. Def. 2})\\ =& \exp (-{\mu }^{\prime })\exp ({\mu }^{\prime }+s^{T}s/2)& & \\ =& \exp (s^{T}s/2),& & \end{array}$$

which is the MGF of a standard normal random variable. Therefore, $W\sim \mathcal{N}(0,I)$.

Case II. $V$ is singular. Then, there exists a vector $a\ne 0$ such that $Va=0$. For simplicity, suppose $\mu =0$. Note that

$$a^{T}Va=\mathbb{E}[(a^{T}X{)}^2]=0.$$

Thus, $a^{T}X=0$, indicating the linear dependence of $X$. WLOG, suppose $X=(Y;Z)\in {\mathbb{R}}^{k=d+(k-d)}$ and the first $0\le d<k$ components of $X$ are linearly independent and $Z=AY$ is a linear combination of $Y$. Since $X$ satisfies the second alternative definition, $Y$ also satisfies it. By the Case I, we know $Y=DW$ for some positive definite $D$ and standard normal $W$. Let $W^{\prime }$ be another $k-d$-dimensional standard normal r.v. independent of $W$. Then, we can write

$$X=\left[\begin{array}{c}Y\\ Z\end{array}\right]=\left[\begin{array}{cc}D& 0\\ AD& 0\end{array}\right]\left[\begin{array}{c}W\\ W^{\prime }\end{array}\right].$$

Properties

• (Sufficiency) The mean and covariance of a multivariate normal distribution consist of a Sufficient Statistic.

  • In other words, the distribution of a multivariate normal random vector is completely determined by its mean and covariance

  • 📎 See Sufficiency for proof

• (Affine transformation). The Affine Transformation of a normal random variable $X$: $a+BX$ is also a normal random variable

  • As a special case, any sub-vector of a normal random vector is also normal

    • As a special case, any component of a normal random vector is also normal

  • If $X\sim \mathcal{N}(\mu ,\Sigma )$, then $BX+a\sim \mathcal{N}(B\mu +a,B\Sigma B^T)$

  • 📎 The proof follows the alternative definition 1 above

• (Independent Gaussians are jointly Gaussian). The sum of independent normal random variables is also a normal random variable

  • 📎 Prove this using the Inversion Theorem

  • ❗️ Note that this is generally not true for dependent random variables

  • ❗️ More generally, if a random vector with normal components is not jointly normal, then its affine transformation is not necessarily normal! See also Independence, Correlation, and Jointly Normal

• (Independent iff Uncorrelated). For a multivariate normal random vector, its components are independent if and only if they are uncorrelated

  • 📎 We can use the sufficiency property to prove this. For $X$ with uncorrelated components, we can construct $Y$ with independent components that have the same mean and variance. Then $X\stackrel{d}{=}Y$ and thus $X$ has independent components. See also Independence, Correlation, and Jointly Normal

  • 📎 For nondegenerate normal random vector, we can also factorize the PDF to show independence.

  • ❗️ The statement is not true for general Random Variables for which the mean and variance are not sufficient; see Independence, Correlation, and Jointly Normal

• Hence, if $X\sim \mathcal{N}(\mu ,{\sigma }^2)$, then $Z=(X-\mu )/\sigma$ is normal with mean 0 and variance 1; $Z$ is said to have a standard or unit normal distribution

  • We write the CDF of a standard normal distribution $\Phi$

• (Symmetry). $\Phi (-x)=1-\Phi (x)$

• Let ${\mathbb{R}}^p\ni X\sim \mathcal{N}(\mu ,\Sigma )$; then, $(X-\mu {)}^T{\Sigma }^{-1}(X-\mu )\sim {\chi }_p^2$ (see Chi-Square Distribution)

Proofs

Sufficiency

Note that PDF or CDF completely determines the distribution of a random variable. Therefore, by the definition for nondegenerate multivariate normal distribution via PDF, we can see that the mean and covariance matrix are sufficient.

For general normal random vectors, we can seek help from the MGF. Note that a MGF also completely determines a distribution. We use the second alternative definition. For a random vector, its MGF is the multivariate transform:

$$M_X(s)=\mathbb{E}[\exp (s^{T}X)]=M_{s^{T}X}(1)=\exp (s^T\mu +s^{T}Vs/2),$$

where we use the definition that $s^{T}X$ is normal with mean $s^T\mu$ and variance $s^{T}Vs$. As we can see, the mean and variance are sufficient to determine the MGF, and thus the distribution.

Independence, Correlation, and Jointly Normal

Normal components does not imply jointly normal.

It is not true that if $X$ and $Y$ are both normal, then the joint distribution of $(X,Y)$ is normal. For example, let $X\sim \mathcal{N}(0,1)$ and $Y=(2B-1)X$, where $B\sim \mathrm{Bern}(0.5)$, i.e., $Y=\pm X$ with equal probability. Then, it is easy to verify that $\Phi$ is also the CDF of $Y$, and thus $Y\sim \mathcal{N}(0,1)$. However, if $(X,Y)$ is jointly normal, we would have $(1,1)(X,Y{)}^T=X+Y$ is normal, which is not true because $X+Y=2BX$.

Independent normal components implies jointly normal.

The above statement becomes true once we impose the independence condition. We use the second alternative definition above to prove this. Let $X=(X_1,\dots ,X_n)$ with normal components. Then, for any vector $a$, we have $a^{T}X={\sum }_{i=1}^n{a}_i{X}_i$. Note that $a_i{X}_i$ are independent normal random variables, and thus their sum is normal by Property ^prop-ind-joint, or the Inversion Theorem. By the alternative definition, $X$ is jointly normal.

Joint normal with zero correlation implies independence.

Suppose that the components of $X$ are uncorrelated, i.e., its covariance matrix is a diagonal. Consider another random vector $Y$ such that $Y_i\stackrel{d}{=}{X}_i$ and $Y_i$ are independent. By Property ^prop-ind-joint, $Y$ is jointly normal. Since $X$ and $Y$ have the same mean and covariance, by Property ^prop-suff, $X$ and $Y$ have the same distribution, and thus the components of $X$ are independent.

Zero correlation does not imply independence for general random variables.

For example, let $X\sim \mathrm{Unif}[-1,1]$ and $Y=X^2$. Certainly, $X$ and $Y$ are not independent, but they are uncorrelated:

$$\mathrm{Cov}(X,Y)=\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]=\mathbb{E}[X^3]-0=0.$$

Sample Mean and Sample Variance

Thm

If $\{X_i{\}}_{i=1}^n$ is a sample from a normal population having mean $\mu$ and variance ${\sigma }^2$, then $\overline{X}$ and $S^2$ are independent random variables, with $\overline{X}\sim \mathcal{N}(\mu ,{\sigma }^2/n)$, and $\frac{n-1}{{\sigma }^2}{S}^2\sim {\chi }_{n-1}^2$. Then we have $\frac{\overline{X}-\mu }{S/\sqrt{n}}\sim t_{n-1}$.

• ❗️ This independence of $\overline{X}$ and $S^2$ is a unique property of the normal distribution.

General Bivariate Normal Distribution

We skipped the introduction of standard bivariate normal random variable $(X,Y)\in {\mathbb{R}}^2$, as it is more convenient, and not conceptually harder, to directly deal with the general bivariate normal distribution.

Let $X$ and $Y$ be two random vectors with a joint normal distribution. That is

$$\left[\begin{array}{c}X\\ Y\end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{c}{\mu }_X\\ {\mu }_Y\end{array}\right],\left[\begin{array}{cc}{\Sigma }_{XX}& {\Sigma }_{XY}\\ {\Sigma }_{YX}& {\Sigma }_{YY}\end{array}\right]\right).$$

Suppose ${\Sigma }_{YY}>0$ (PSD). We have

• $\mathbb{E}[X\mid Y]={\mu }_X+{\Sigma }_{XY}{\Sigma }_{YY}^{-1}(Y-{\mu }_Y)$.
• $\tilde{X}:=X-\mathbb{E}[X\mid Y]$ is independent of $Y$, and thus independent of any function of $Y$.
• $\mathrm{Cov}(\tilde{X},\tilde{X}\mid Y)=\mathrm{Cov}(\tilde{X},\tilde{X})={\Sigma }_{XX}-{\Sigma }_{XY}{\Sigma }_{YY}^{-1}{\Sigma }_{YX}$.

Before proving the above results, we recover the standard bivariate normal distribution from them. Let ${\Sigma }_{XX}={\sigma }_X^2$, ${\Sigma }_{YY}={\sigma }_Y^2$, ${\Sigma }_{XY}=\rho {\sigma }_X{\sigma }_Y$. Then, we have

$$\mathbb{E}[X\mid Y]=\rho \frac{{\sigma }_X}{{\sigma }_Y}Y,\mathrm{Var}(\tilde{X})={\sigma }_X^2(1-{\rho }^2).$$

Proofs

WLOG, we assume ${\mu }_X={\mu }_Y=0$ for simplicity. Let $\hat{X}={\Sigma }_{XY}{\Sigma }_{YY}^{-1}Y$. We have

$$\mathbb{E}[\hat{X}Y]={\Sigma }_{XY}{\Sigma }_{YY}^{-1}\mathbb{E}[YY]={\Sigma }_{XY}=\mathbb{E}[XY]⟹\mathbb{E}[(X-\hat{X})Y]=0.$$

Note that $(X-\hat{X},Y)$ is a linear transformation of $(X,Y)$, and thus the above uncorrelatedness implies independence. This also implies, for any function $g$:

$$\mathbb{E}[(X-\hat{X})g(Y)]=0.$$

By the General Definition of Conditional Expectation, the above condition says that

$$\mathbb{E}[X\mid Y]=\hat{X}.$$

Finally, we compute $\mathrm{Var}(\tilde{X})$. Since $\mathbb{E}[X\mid Y]$ is also a function of $Y$, it is independent of $\tilde{X}$. Thus, we have

$$\mathrm{Var}(\tilde{X})=\mathbb{E}\left[(X-\hat{X})(X-\hat{X}{)}^T\right]=\mathbb{E}\left[(X-\hat{X})X^T\right]={\Sigma }_{XX}-{\Sigma }_{XY}{\Sigma }_{YY}^{-1}{\Sigma }_{YX}.$$