f-Divergence

For two distributions $P$ and $Q$ such that $Q>0$ whenever $P>0$, and a convex real function $f$ on $[0,+\infty )$ with $f(1)=0$ and $f(0)={\lim }_{t\to 0+}f(t)$, we can define

$$D_f(P\Vert Q)=\int f\left(\frac{\mathrm{d}P(x)}{\mathrm{d}Q(x)}\right)\mathrm{d}Q(x)\mathrm{d}x={\mathbb{E}}_{q}f(p/q),$$

where the second equality holds when $p$ and $q$ are densities of $P$ and $Q$ respectively. $D_f(P\Vert Q)$ is called the f-divergence between $P$ and $Q$.

Intuitively, the expected convex function ${\mathbb{E}}_{Q}f$ amplifies the difference signal in the likelihood ratio $\mathrm{d}P/\mathrm{d}Q$.

Examples

• Letting $f(x)=1/2|x-1|$ gives the Total Variation Distance
• Letting $f(x)=x\log x$ gives the KL Divergence
• Letting $f(x)=1/2(\sqrt{t}-1{)}^2$ gives the squared Hellinger Distance
• Letting $f(x)=(x-1{)}^2$ gives the chi-square divergence

TV vs. KL ^rmk-tv-kl

• TV is a metric, meaning that it’s symmetric and satisfies the triangle inequality; KL is not a metric, and does not satisfy these two properties.
• TV does no “tensorize” but KL does. So in practice, KL, or other divergences that tensorize, is usually preferred, as the calculation boils down to calculating the divergence of individual samples.
• TV is convenient for theoretical analysis, as it has a simple variational representation, L1 norm equivalence, and optimal transportation interpretation. For example, TV is used to show hardness results of hypothesis testing.

Properties

• Non-negativity: $D_f(P\Vert Q)\ge 0$, due to the convexity of $f$.
• Zero self-divergence: $D_f(P\Vert P)=0$, due to that $f(1)=0$.