Evaluating a Test

Since a Hypothesis Testing task is a binary decision-making problem, we have two basic metrics for evaluating a test $\psi$:

• Type I error, a.k.a false positive rate, is the probability of rejecting $H_0$ when it is true: $P_{{\theta }_0}(\psi (X)=1)$.
• Type II error, a.k.a false negative rate, is the probability of failing to reject $H_0$ when it is false: $P_{{\theta }_1}(\psi (X)=0)$.

More generally, Type I and II errors are functions indexed by the test $\psi$, defined as

$$\{\begin{array}{llcc}{\alpha }_{\psi }& :{\Theta }_0\to \mathbb{R},& {\theta }_0\mapsto P_{{\theta }_0}(\psi (X)=1),& \text{(Type I error)}\\ {\beta }_{\psi }& :{\Theta }_1\to \mathbb{R},& {\theta }_1\mapsto P_{{\theta }_1}(\psi (X)=0),& \text{(Type II error)}\end{array}$$

The largest Type I error is called the size of the test:

$${\overline{\alpha }}_{\psi }=\underset{{\theta }_0\in {\Theta }_0}{\sup }{\alpha }_{\psi }({\theta }_0).$$

Recall that in Hypothesis Testing, we use data to disprove the null. Thus, size evaluates how confident the test is. For a test of size $\alpha$, it correctly fails to reject the null with an $(1-\alpha )$ confidence. See also Confidence Interval and Hypothesis Test Duality.

We say a test has (asymptotic) level $\alpha$ if its size is at most $\alpha$:

$$\underset{n\to \infty }{\lim }\underset{{\theta }_0\in {\Theta }_0}{\sup }{\alpha }_{\psi }({\theta }_0)\le \alpha .$$

A small size ensures that, under the null, the test does not reject the null with high probability. The power of a test describes, under the alternative, its ability to correctly reject the null:

$${\pi }_{\psi }=\underset{{\theta }_1\in {\Theta }_1}{\inf }(1-{\beta }_{\psi }({\theta }_1)).$$

We summarize the above metrics into a confusion matrix:

Truth \ Decision	Fail to Reject	Reject
$H_0$	True Negative Confidence $1-\alpha$	False Positive Type I Error Size Significance Level $\alpha$
$H_1$	False Negative Type II Error $\beta$	True Positive Power $1-\beta$

Balanced Evaluation

Different situations favors different evaluation metrics. Specifically, one may want to balance the Type I and Type II errors, or focus more on one of them. There are two common ways to balance the two.

Bayes Risk

Bayes risk assigns different weights to the Type I and Type II errors, according to the loss function and prior. Specifically, the Bayes risk is the weighted sum:

$$R_B(\psi ,\pi )=\alpha \cdot {\pi }_0{c}_{\mathrm{FP}}+\beta \cdot {\pi }_1{c}_{\mathrm{FN}},$$

where $\pi$ is the prior for $\theta$ and $c_{\mathrm{FP}}$ and $c_{\mathrm{FN}}$ are the costs of Type I and Type II errors, respectively. See Bayes Optimal Test for the optimal test under this metric.

Neyman-Pearson Paradigm

Neyman-Pearson paradigm, or Uniformly Most Powerful Testing for general HT, for a simple alternative hypothesis formulates the problem as a constrained optimization problem:

$$\begin{array}{rl}\underset{\psi }{\max }& {\pi }_{\psi }\\ \text{s.t.}& {\alpha }_{\psi }({\theta }_0)\le \alpha ,\forall {\theta }_0\in {\Theta }_0,\end{array}$$

which is equivalent to

$$\begin{array}{rl}\underset{\psi }{\min }& {\beta }_{\psi }({\theta }_1)\\ \text{s.t.}& \underset{{\theta }_0\in {\Theta }_0}{\max }{\alpha }_{\psi }({\theta }_0)\le \alpha .\end{array}$$

It puts the Type I error as a size constraint, and maximizes the power (minimizes the Type II error) under this constraint. Similarly, this reflects the asymmetry between the null and alternative hypotheses (^d85be2). See Uniformly Most Powerful Test for the optimal test under this metric.