Evaluating a Test

Since a Hypothesis Testing task is a binary decision-making problem, we have two basic metrics for evaluating a test $\psi$:

• Type I error, a.k.a false positive rate, is the probability of rejecting $H_0$ when it is true: $P_{{\theta }_0}(\psi (X)=1)$.
• Type II error, a.k.a false negative rate, is the probability of failing to reject $H_0$ when it is false: $P_{{\theta }_1}(\psi (X)=0)$.

More generally, Type I and II errors are functions indexed by the test $\psi$, defined as

$$\{\begin{array}{llcc}{\alpha }_{\psi }& :{\Theta }_0\to \mathbb{R},& {\theta }_0\mapsto P_{{\theta }_0}(\psi (X)=1),& \text{(Type I error)}\\ {\beta }_{\psi }& :{\Theta }_1\to \mathbb{R},& {\theta }_1\mapsto P_{{\theta }_1}(\psi (X)=0),& \text{(Type II error)}\end{array}$$

The largest Type I error is called the size of the test:

$${\overline{\alpha }}_{\psi }=\underset{{\theta }_0\in {\Theta }_0}{\sup }{\alpha }_{\psi }({\theta }_0).$$

Recall that in Hypothesis Testing, we use data to disprove the null. Thus, size evaluates how confident the test is. For a test of size $\alpha$, it correctly fails to reject the null with an $(1-\alpha )$ confidence. See also Confidence Interval and Hypothesis Test Duality.

We say a test has (asymptotic) level $\alpha$ if its size is at most $\alpha$:

$$\underset{n\to \infty }{\lim }\underset{{\theta }_0\in {\Theta }_0}{\sup }{\alpha }_{\psi }({\theta }_0)\le \alpha .$$

A small size ensures that, under the null, the test does not reject the null with high probability. The power of a test describes, under the alternative, its ability to correctly reject the null:

$${\pi }_{\psi }=\underset{{\theta }_1\in {\Theta }_1}{\inf }(1-{\beta }_{\psi }({\theta }_1)).$$

We summarize the above metrics into a confusion matrix:

Truth \ Decision	Fail to Reject	Reject
$H_0$	True Negative Confidence $1-\alpha$	False Positive Type I Error Size Significance Level $\alpha$
$H_1$	False Negative Type II Error $\beta$	True Positive Power $1-\beta$

FPR and FDR

Don’t confuse the false positive rate (FPR) with the false discovery rate (FDR): the former is $\mathrm{FP}/(\mathrm{FP}+\mathrm{TN})$ and the latter is $\mathrm{FP}/(\mathrm{FP}+\mathrm{TP})$. That is, FPR is FP rate with respect to the ground truth (true nulls), while FDR is FP rate with respect to the test result (rejected nulls).

Balanced Evaluation

Different situations favors different evaluation metrics. Specifically, one may want to balance the Type I and Type II errors, or focus more on one of them. There are two common ways to balance the two.

Bayes Risk

Bayes risk assigns different weights to the Type I and Type II errors, according to the loss function and prior. Specifically, the Bayes risk is the weighted sum:

$$R_B(\psi ,\pi )=\alpha \cdot {\pi }_0{c}_{\mathrm{FP}}+\beta \cdot {\pi }_1{c}_{\mathrm{FN}},$$

where $\pi$ is the prior for $\theta$ and $c_{\mathrm{FP}}$ and $c_{\mathrm{FN}}$ are the costs of Type I and Type II errors, respectively. See Bayes Optimal Test for the optimal test under this metric.

Neyman-Pearson Paradigm

Neyman-Pearson paradigm, or Uniformly Most Powerful Testing for general HT, for a simple alternative hypothesis formulates the problem as a constrained optimization problem:

$$\begin{array}{rl}\underset{\psi }{\max }& {\pi }_{\psi }\\ \text{s.t.}& {\alpha }_{\psi }({\theta }_0)\le \alpha ,\forall {\theta }_0\in {\Theta }_0,\end{array}$$

which is equivalent to

$$\begin{array}{rl}\underset{\psi }{\min }& {\beta }_{\psi }({\theta }_1)\\ \text{s.t.}& \underset{{\theta }_0\in {\Theta }_0}{\max }{\alpha }_{\psi }({\theta }_0)\le \alpha .\end{array}$$

It puts the Type I error as a size constraint, and maximizes the power (minimizes the Type II error) under this constraint. Similarly, this reflects the asymmetry between the null and alternative hypotheses (^d85be2). See Uniformly Most Powerful Test for the optimal test under this metric.