Expectation

• The expectation or mean of a discrete Random Variable is

$$\mathbb{E}[X]=\sum _i{x}_{i}p(x_i),$$

where $\{x_i\}$ is the range of $X$, and $p$ is the PMF of $X$.

• For a continuous Random Variable,

$$\mathbb{E}[X]={\int }_{\mathbb{R}}xf(x)d,$$

where $f$ is the PDF of $X$.

We say $X$ is integrable if $\mathbb{E}[|X|]<\infty$. If one of the expectations $\mathbb{E}[X_{+}]$ or $\mathbb{E}[X_{-}]$ is finite, the expectation is well-defined. Otherwise, the expectation is undefined.

• For nonnegative r.v. $X$,

$$\mathbb{E}[X]={\int }_0^{\infty }P(X>x)dx.$$

• Specifically, if $X$ takes vales in $\mathbb{N}$,

$$\mathbb{E}[X]=\sum _{n\ge 0}P(X>n).$$

• Proof: ![[6-7700-hw4#Exercise 2#Solution|n-h]]
• If $X$ is continuous with Density $f$, the proof is easier: ${\int }_0^{\infty }P(X>x)\mathrm{d}x={\int }_0^{\infty }{\int }_x^{\infty }f(t)\mathrm{d}t\mathrm{d}x={\int }_0^{\infty }{\int }_0^{t}f(t)\mathrm{d}x\mathrm{d}t={\int }_0^{\infty }tf(t)\mathrm{d}t=\mathbb{E}[X],$ where the interchange of the order of integration is justified by Fubini’s Theorem as $f$ is nonnegative.
• For general r.v. $X$, following Abstract Integration, the expectation is defined as

$$\mathbb{E}[X]={\int }_{\mathbb{R}}x\mathrm{d}{P}_X={\int }_{\Omega }X\mathrm{d}P.$$

Properties

• (Linearity) For any linear transformation on random vector $X$, $\mathbb{E}[AX+b]=A\mathbb{E}[X]+b$.
  • Special case: $\mathbb{E}[aX+bY]=a\mathbb{E}[X]+b\mathbb{E}[Y]$.
• (Independence) For independent r.v. $X,Y$, $\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]$.
• (Function transformation) For any measurable function $g$, $\mathbb{E}[g(X)]=\int g(x)\mathrm{d}P(x)$.

Proof of Function Transformation

Let $g=g_{+}-g_{-}$. Using the definition of expectation for nonnegative function and linearity, we have

$$\mathbb{E}[g(X)]=\mathbb{E}[g_{+}(X)]-\mathbb{E}[g_{-}(X)]={\int }_0^{\infty }P(g_{+}(X)>x)\mathrm{d}x-{\int }_0^{\infty }P(g_{-}(X)>x)\mathrm{d}x={\int }_0^{\infty }P(g(X)>x)\mathrm{d}x-{\int }_0^{\infty }P(g(X)<-x)\mathrm{d}x.$$

For the first term on the RHS, we have

$${\int }_0^{\infty }P(g(X)>t)\mathrm{d}t={\int }_0^{\infty }{\int }_{\{x:g(x)>t\}}\mathrm{d}P(x)\mathrm{d}t={\int }_{\mathbb{R}}{\int }_0^{g_{+}(x)}\mathrm{d}t\mathrm{d}P(x)={\int }_{\mathbb{R}}{g}_{+}(x)\mathrm{d}P(x),$$

where the interchange of the order of integration is justified by Fubini’s Theorem. Similarly, the second term is ${\int }_{\mathbb{R}}{g}_{-}(x)\mathrm{d}P(x)$. Therefore, we have

$$\mathbb{E}[g(X)]=\int g_{+}(x)\mathrm{d}P(x)-\int g_{-}(x)\mathrm{d}P(x)=\int g(x)\mathrm{d}P(x).$$