Convergence of Random Variables

Relation between Convergence Modes

$$\begin{array}{cc}& X_n\stackrel{\text{a.s.}}{\to }X\\ & \text{or}\\ X_n\stackrel{L_{p^{\prime }}}{\to }X\stackrel{p^{\prime }\ge p}{⟹}& X_n\stackrel{L^p}{\to }X\end{array}⟹X_n\stackrel{P}{\to }X⟹\begin{array}{c}{f}_{X_n}\to f_X\\ \Downarrow \\ X_n\stackrel{d}{\to }X\\ \Updownarrow \\ p_{X_n}\to p_X\end{array}⟺{\varphi }_{X_n}\to {\varphi }_X,$$

where $f$, $p$, and $\varphi$ are PDF (for continuous r.v.s), PMF (for discrete r.v.s), and characteristic function, respectively, and their convergences are in the sense of convergence of functions: $g_n(t)\to g(t)\forall t$.

• 📎 If $X$ follows a point-mass/Dirac Distribution ${\delta }_c$, then $X_n\stackrel{d}{\to }X⟹X_n\stackrel{P}{⟶}X$.

Convergence under transformations

Mode \ Operation	CMT (g) 1	Addition (+)	Multiplication (×)	Division (÷) 2	Joint Distribution (·,·)
$X_n\stackrel{d}{\to }X,Y_n\stackrel{d}{\to }c$	✓	✓	✓	✓	✓
$X_n\stackrel{d}{\to }X,Y_n\stackrel{d}{\to }Y$	✓
$X_n\stackrel{P}{\to }X,Y_n\stackrel{P}{\to }Y$	✓	✓	✓	✓	✓
$X_n\stackrel{\text{a.s.}}{\to }X,Y_n\stackrel{\text{a.s.}}{\to }Y$	✓	✓	✓	✓	✓
$X_n\stackrel{L^p}{\to }X,Y_n\stackrel{L^p}{\to }Y$		✓			✓

As a Random Variable involves many different elements, we can define various modes of convergence for a sequence of random variables. Some definitions view random variables as Measurable functions, others as probability measures, and some through their associated functions, such as the CDF and the Characteristic Function.

In this note, we denote a sequence of random variables as $X_n$.

Almost Sure/ Strong Convergence

• Definition: We say that $X_n$ converges almost surely/ almost everywhere/ with probability 1/ strongly to $X$ if

$$\mathbb{P}(\underset{n\to \infty }{\lim }{X}_n=X)=1.$$

• Notation: $X_n\stackrel{\text{a.s.}}{⟶}X$.
• Alternative interpretation: Viewing a random variable as a measure of events, the above convergence is equivalent to

$$\mathbb{P}(\omega \in \Omega :\underset{n\to \infty }{\lim }{X}_n(\omega )=X(\omega ))=1,$$

where $\Omega$ is the event space.

• ❗️ Remark: $X_n$ are generally highly dependent for this convergence to hold.

  • 📗 Partial sum → infinite sum

  • 📗 R.v.s defined as converging functions of a single underlying r.v.

Convergence in Probability

• Definition: We say that $X_n$ converges in probability to $X$ if for any $ϵ>0$,

$$\underset{n\to \infty }{\lim }\mathbb{P}(|X_n-X|>ϵ)=0.$$

• Notation: $X_n\stackrel{P}{⟶}X$.

• ❗️ Remark: Convergence in probability has a similar interpretation as almost sure convergence, and it also generally requires to be dependent. However, it’s weaker than almost sure convergence as it’s not uniform: different $(ϵ,\delta )$ pairs such that $\mathbb{P}(|X_n-X|>ϵ)\le \delta$ require different $n$.

Convergence in Distribution/ Weak Convergence

• Definition: We say that $X_n$ converges in distribution/ in law/ weakly to $X$ if

$$\underset{n\to \infty }{\lim }{F}_{X_n}(x)=F_X(x),$$

for all $x$ at which $F_X$ is continuous, where $F$ is a Cumulative Distribution Function.

• Notation: $X_n\stackrel{d}{⟶}X$.

• Alternative interpretation: Weak convergence inspects the convergence of the associated CDF of a random variable sequence, and thus it’s weaker and generally requires no dependency between $X_n$. It’s equivalent to the convergence of the Characteristic Function.

• ❗️ Remark: Weak convergence is consistent with the convergence of real numbers. If $X_n\stackrel{\text{a.s.}}{=}{a}_n\in \mathbb{R}$ and $X\stackrel{\text{a.s.}}{=}a\in \mathbb{R}$, then $X_n\stackrel{d}{\to }X⟺a_n\to a$.

  • ❗️ This consistency does not hold if we require $F_{X_n}\to F_X$ for all $x$.

Portmanteau Lemma

Several important statements equivalent to convergence in distribution are given by the Portmanteau Lemma:

1. $\mathbb{E}g(X_n)\to \mathbb{E}g(X)$ for any bounded, continuous/Lipschitz function $g$.
2. ${\mathrm{liminf}}_{n\to \infty }\mathbb{E}g(X_n)\ge \mathbb{E}g(X)$ for any nonnegative and continuous function.
3. ${\mathrm{liminf}}_{n\to \infty }P(X_n\in B)\ge P(X\in B)$ for any open set $B$.
4. ${\mathrm{limsup}}_{n\to \infty }P(X_n\in B)\le P(X\in B)$ for any closed set $B$.
5. $P(X_n\in B)\to P(X\in B)$ for any continuity set³ $B$.
   • which is further equivalent to $\left|{\int }_B{f}_{X_n}(x)-f_X(x)\right|\mathrm{d}x\to 0$.

Convergence of PDF/PMF

Suppose $X_n\stackrel{d}{\to }X$. The convergence of the associated PDF/PMF is unclear:

• It is possible for $X_n$ to be discrete and $X$ to be continuous.

  • 📗 $X_n=\frac{1}{n}\mathrm{Unif}\{1,\dots ,n\}\to \mathrm{Unif}[0,1]$.

• It is possible for $X_n$ to be continuous and $X$ to be discrete.

  • 📗 $X_n=\mathrm{Unif}[0,1/n]\to 0$.

• ❗️ If $X_n$ and $X$ are continuous, it is possible that the PDF $f_n\nrightarrow f$ does not converge.

  • 📗 $F_n=x+\cos (2\pi nx)/(2\pi n)\to F_X=x$ but $f_n=1-\sin (2\pi nx)\nrightarrow f_X=1$

For the other direction, we have:

• If $X_n$ and $X$ are continuous, the convergence of PDF implies the convergence in distribution.
• If $X_n$ and $X$ are discrete, the convergence in distribution is equivalent to the convergence of PMF.

Convergence of Characteristic Functions

Convergence in distribution is equivalent to the convergence of Characteristic Functions:

$$\underset{n\to \infty }{\lim }{\varphi }_{X_n}(t)={\varphi }_X(t),\forall t$$

Convergence in $L^p$ Norm

• Definition: We say that $X_n$ converges ==in $L^p$ norm/ in $p$th mean== to $X$ if

$$\underset{n\to \infty }{\lim }\mathbb{E}[|X_n-X{|}^p]=0.$$

• Notation: $X_n\stackrel{L^p}{⟶}X$.

• ❗️ Remark: For $p_1>p_2\ge 1$, we have $X_n\stackrel{L_{p_1}}{⟶}X⟹X_n\stackrel{L_{p_2}}{⟶}X$, but not the other way around.

Convergence under Transformations

Continuous Mapping Theorem

Let $X_n$ be a sequence of random variables that converges almost surely/ in probability/ in distribution to $X$. Let $g$ be a continuous function. Then $g(X_n)$ converges to $g(X)$.

• ❗️ The continuous mapping theorem does not apply to convergence in norm.

Further, if $g$ is continuous at $c$ and $X_n\stackrel{P}{\to }c$, then $g(X_n)\stackrel{P}{\to }g(c)$. $g$ need not be continuous in this case.

Slutsky’s Theorem

Let $X_n$ and $Y_n$ be sequences of random variables that converge in distribution to $X$ and $c$ respectively, where $c$ is a constant. Then

$$\mathcal{B}(X_n,Y_n)\stackrel{d}{\to }\mathcal{B}(X,c),$$

where $\mathcal{B}=\{+,\times ,÷,(\cdot ,\cdot )\}$ is a set of binary operations. For $÷$ to be defined, $Y_n,c$ must be nonzero.

• ❗️ The theorem also holds for convergence in probability.

For almost sure convergence, convergence in probability, and in $L^p$ norm, we have stronger results:

Suppose $X_n\stackrel{\text{a.s}/P}{\to }X$ and $Y_n\stackrel{\text{a.s}/P}{\to }Y$. Then

$$\mathcal{B}(X_n,Y_n)\stackrel{\text{a.s.}/P}{\to }\mathcal{B}(X,Y).$$

Suppose $X_n\stackrel{L^p}{\to }X$ and $Y_n\stackrel{L^p}{\to }Y$. Then

$$X_n+Y_n\stackrel{L^p}{\to }X+Y.$$

Note that we no longer restrict $Y_n$ to converge to a constant.

Sum of IID Random Variables

A good example to illustrate different modes of convergence is the sum, or average, of iid random variables. Suppose $X_i$ are iid with finite mean $\mu$ and variance ${\sigma }^2$. Let ${\overline{X}}_n={\sum }_{i=1}^n{X}_i$. Then, the following theorems claim that ${\overline{X}}_n$ converges to $\mu$,

• in distribution by Central Limit Theorem;
• in probability by Weak Law of Large Numbers;
• and almost surely by Strong Law of Large Numbers.

Footnotes

1. The function $g$ is applied on $Y_n$ and $Y$, and is continuous on the range of $Y$. ↩

2. For the division operation to be well-defined, the denominator sequence and its limit must be non-zero. ↩

3. A continuity set has a zero-measure boundary. ↩