Covariance

The covariance of two Random Variables is

$$\mathrm{Cov}(X,Y)=\mathbb{E}[(X-{\mu }_x)(Y-{\mu }_y)]=\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y].$$

More generally, for two random vectors $X,Y$,

$$\mathrm{Cov}(X,Y)=\mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y]{)}^T].$$

Covariance as an Inner Product

Covariance is like an Inner Product (it is indeed an inner product for centralized Random Variables). We have

• $⟨X,X⟩\ge 0$ and $⟨X,X⟩=0$ iff $X=C$ (semi-positiveness)
• $⟨X,Y⟩=⟨Y,X⟩$
• $⟨X,Y+b⟩=⟨X,Y⟩$ where $b$ is a constant
• $⟨{\sum }_i{a}_i{X}_i,{\sum }_j{b}_j{Y}_j⟩={\sum }_i{\sum }_j{a}_i{b}_j⟨X_i,Y_j⟩$
  • Generalization: $⟨AX,BY⟩=A⟨X,Y⟩B^T$
• Independence (orthogonal) ⇒ $⟨X,Y⟩=0$
  • However, zero covariance doesn’t imply independence.
• Cauchy-Schwarz Inequality: $|⟨X,Y⟩|\le \sqrt{⟨X,X⟩⟨Y,Y⟩}=\sqrt{\mathrm{Var}(X)}\sqrt{\mathrm{Var}(Y)}$, and the equality holds iff $Y\stackrel{\text{a.e.}}{=}cX$ for some constant $c$.

Correlation

$$\mathrm{Corr}(X,Y)=\frac{\mathrm{Cov}(X,Y)}{\sqrt{\mathrm{Var}(X)\mathrm{Var}(Y)}}$$

Denote $\rho =\mathrm{Corr}(X,Y)$. Then $-1\le \rho \le 1$, and $\rho =\pm 1$ iff $Y=aX+b$ a.s. for some $a\gtrless 0$.

Sample Covariance

For two samples of $X,Y$ of size $n$, their sample covariance is

$$\frac{{\sum }_{i=1}^n(x_i-\overline{x})(y_i-\overline{y})}{n-1}$$

For centered random vectors, we have

$$\Sigma =\frac{1}{n-1}\sum _{i=1}^n{x}_i{y}_i^T=\frac{1}{n-1}{X}^{T}Y,$$

where $X=(x_1,\dots ,x_n{)}^T$.

Sample Correlation Coefficient

For two samples of $X,Y$ of size $n$, their sample correlation coefficient is

$$r=\frac{{\sum }_{i=1}^n(x_i-\overline{x})(y_i-\overline{y})}{(n-1)s_x{s}_y}=\frac{{\sum }_{i=1}^n(x_i-\overline{x})(y_i-\overline{y})}{\sqrt{{\sum }_{i=1}^n(x_i-\overline{x}{)}^2}\sqrt{{\sum }_{i=1}^n(y_i-\overline{y}{)}^2}},$$

where $s_x,s_y$ is the sample Variance of $\{x_i\},\{y_i\}$.

Properties

When $r>0$, we say the sample data pairs are positively correlated; When $r<0$, we say the sample data pairs are negatively correlated.

$r$ have some properties:

• $r\in [-1,1]$
• $r=1$ if $y_i=a{x}_i+b$ and $a>0$
• $r=-1$ if $y_i=a{x}_i+b$ and $a<0$
• if $r$ is the sample correlation coefficient for $\{(x_i,y_i)\}$, then it is also the sample correlation coefficient for $\{(a{x}_i+b,c{y}_i+d)\}$ provided that $a,c$ have the same sign.