Cauchy-Schwarz Inequality

In Vector Space

For vectors $x,y$ in an Inner Product space,

$$|⟨x,y⟩{|}^2\le ⟨x,x⟩⟨y,y⟩,$$

or equivalently,

$$|⟨x,y⟩|\le \Vert x\Vert \Vert y\Vert ,$$

where the norm is induced by the inner product. The equality holds if and only if $x$ and $y$ are linearly dependent.

In Probability

For Random Variables $X$, $Y$ with a finite second moment,

$$\mathbb{E}[|XY|]\le \sqrt{\mathbb{E}[X^2]}\sqrt{\mathbb{E}[Y^2]}.$$

The equality holds if and only if $X=aY$ for some constant $a$. This is a special case of In Vector Space as Expectation is just the $L^2$ inner product w.r.t the Probability Measure.

An application of this is Covariance as an Inner Product.

Proofs

First consider the probability space. WLOG, consider nonnegative r.v.s. If $\mathbb{E}[Y^2]=0$, then $Y=0$ a.s., and the inequality is trivial. Otherwise, let $a=\mathbb{E}[XY]/\mathbb{E}[Y^2]$. Then

$$\begin{array}{rl}0& \le \mathbb{E}[(X-aY{)}^2]=\mathbb{E}[X^2]-2a\mathbb{E}[XY]+a^2\mathbb{E}[Y^2]\\ & =\mathbb{E}[X^2]-\frac{(\mathbb{E}[XY]{)}^2}{\mathbb{E}[Y^2]}.\end{array}$$

From the inequality, the equality holds iff $X=aY$ a.s. We can find $a$ in a more principled way.

Quadratic

Consider the quadratic:

$$\begin{array}{rl}0\le ⟨x+ty,x+ty⟩& =\sum (x_i+t{y}_i{)}^2\\ & =\left(\sum y_i^2\right)t^2+2\left(\sum x_i{y}_i\right)t+\left(\sum x_i^2\right)\\ ⟹& {\left(\sum x_i{y}_i\right)}^2\le \left(\sum x_i^2\right)\left(\sum y_i^2\right).\end{array}$$

On a complex field, we replace $t$ with $t\alpha$ where $t\in \mathbb{R}$, $\alpha ⟨x,y⟩=|⟨x,y⟩|$.

Projection

We decompose $y$ as its projection onto the direction of $x$ and an orthogonal part $z$. We have

$$0\le \Vert z{\Vert }^2={\Vert y-\frac{⟨x,y⟩}{⟨x,x⟩}x\Vert }^2=\Vert y{\Vert }^2-\frac{⟨x,y{⟩}^2}{\Vert x{\Vert }^2}$$

which implies the result.