Hypothesis Testing

Takeaway Card

• Numerous concepts around hypothesis testing (HT) can be confusing. Always locate yourself in the general statistical binary decision-making framework.
• Under this framework, the question asked for the task is “how to evaluate a test?” All different metrics stem from the two basic ones: Type I error and Type II error. See also the confusion matrix of an HT. Remember that your evaluation can always balance the two errors, or focus on one of them.
• The question asked for the algorithm is “how to construct a test?” This statistical procedure is simply a transformation of the sample into a binary decision rule. One transformation path is through test statistic and rejection region. Another path is through test statistic and p-value.
• HT focuses on disproving the null hypothesis, resulting in an asymmetry between the null and alternative hypotheses. Calculating the Type I error, test statistic (under the null), and p-value only requires the null hypothesis. However, the alternative hypothesis plays a role in shaping the belief about the complement of the null and dictating the direction of extremeness.

Hypothesis testing (HT) is a classical statistical decision-making problem, and can be extended to more general binary statistical decision-making problems. Given sample $X$, we need to make a decision $A(X)$ such that $A(X)\approx 1_{H_1}$, where $H_1$ is the alternative hypothesis. In the context of HT, the statistical procedure $A$ is often called a test, and denoted as $\psi (X)$. A test is a Statistic.

Formally, given a Statistical Model $\{P_{\theta }{\}}_{\theta \in \Theta }$, we want to test the following hypotheses:

$$\{\begin{array}{ll}{H}_0:\theta \in {\Theta }_0,& \text{(null hypothesis)}\\ H_1:\theta \in {\Theta }_1,& \text{(alternative hypothesis)}\end{array}$$

where ${\Theta }_0$ and ${\Theta }_1$ are disjoint subsets of $\Theta$.

Basic Concepts

• Asymmetry in $H_0$ and $H_1$: the data is only used to try to disprove $H_0$. The result of an HT is either to reject or fail to reject the null hypothesis $H_0$.
• If ${\Theta }_0\cup {\Theta }_1=\Theta$, then we say we test $H_0$ against $H_1$. In this case, rejecting $H_0$ implies acceptance of $H_1$.
• Failing to reject $H_0$ never implies acceptance of $H_0$, but only that we do not have enough evidence to reject it.
• When ${\Theta }_0$ and ${\Theta }_1$ are singletons, we call it a simple-simple HT. Otherwise, we call it a composite HT.
• Suppose ${\Theta }_0=\{{\theta }_0\}$ and ${\theta }_0\in \mathbb{R}$. Then the HT is two-sided if $H_1:\theta \ne {\theta }_0$, or is one-sided if $H_1:\theta <{\theta }_0$ or $H_1:\theta >{\theta }_0$.

How to Evaluate a Test

We now focus on the test, i.e., the statistical procedure/algorithm/policy $\psi$ for an HT. The first question is

• ❓ What is a good/optimal test?

Please refer to Evaluating a Test for some answers.

How to Construct a Test

Once we determine the evaluation criteria for a test, the next question is

• ❓ How to construct a test $\psi$ that satisfies the criteria?

In this note, we focus on constructing tests that achieve a certain significance level $\alpha$.

flowchart LR
subgraph B[Transformation]
    BA1[test statistic]; BA2[rejection region]
    B1[?]; B2[?];
    BB1[test statistic]; BB2[p-value]
    BC1[confidence interval]; BC2[rejection region]
end
A --- BA1 --- BA2 --- C
A --- BB1 --- BB2 --- C
A --- BC1 --- BC2 --- C
A[sample] --- B1 --- B2 --- C[test]

Simple constructions directly map the sample to a decision rule, for example: after tossing a coin 4 times, we decide the coin is biased (towards heads) if the number of heads is greater than 2.

More sophisticated and principled methods are needed. In response, some transformations of the sample (statistics) are introduced to construct the test. We discuss two examples:

• Test Statistic and Rejection Region. A test statistic is a statistic of the sample usually with a known distribution under the null hypothesis. Then the critical values form a rejection region for the test statistic. The test is then based on whether the test statistic falls into the rejection region.
• p-value. Sometimes critical values are not available, or the rejection region is not easy to construct. The use of p-value eliminates the need for rejection regions. p-value is a statistic of the test statistic (which is a random variable). The test is then based on whether the p-value is smaller than the level $\alpha$.
  • If we treat p-value as the test statistic, we can see that it gives a principled way of constructing rejection regions: $\mathrm{RR}=\{p\le \alpha \}$, without the need for other critical values.

Test Statistic and Rejection Region

For a hypothesis and sample $\{x_i\}$, we construct a rejection region of the following form:

$$\mathrm{RR}=\{x_1,\dots ,x_n\mid t\ge c\},$$

where $t$ is called the test statistic, and $c$ is called the critical value. If $\{x_i\}\in \mathrm{RR}$ , we reject the hypothesis.

See CLT Test Statistic for an example of a test statistic.

Rejection Region by Confidence Interval

There is a duality between confidence interval and hypothesis tests. Suppose we have a level $(1-\alpha )$ Confidence Interval for $\theta$ given by $[l(\mathbf{x}),u(\mathbf{x})]$. Then the rule “reject ${\mathrm{H}}_0:\theta ={\theta }_0$ if ${\theta }_0\notin [l(\mathbf{x}),u(\mathbf{x})]$” has a significance level $\alpha$:

$$P_{{\theta }_0}(\psi (X)=1)=P_{{\theta }_0}\left([l(X),u(X)]\not\ni {\theta }_0\right)=\alpha .$$

Therefore, a rejection region can be constructed by the complement of the confidence interval. See Confidence Interval and Hypothesis Test Duality for constructing CIs from HTs.

Rejection Region by Likelihood Ratio

Rejection Region

We can also construct a rejection region using the Likelihood ratio:

$$\Lambda (\mathbf{x})=\frac{{\sup }_{\theta \in {\Theta }_0}L(\theta \mid \mathbf{x})}{{\sup }_{\theta \in \Theta }L(\theta \mid \mathbf{x})}=\frac{L({\hat{\theta }}_0)}{L({\hat{\theta }}_{\mathrm{MLE}})},$$

where ${\Theta }_0$ is the null hypothesis parameter space. To highlight the role of the alternative, we can also constrain the supremum to the alternative parameter space ${\Theta }_1$ in the denominator. By definition, the maximizers are called constrained MLEs.

Then, the rejection region is given by

$$\mathrm{RR}=\{\mathbf{x}\mid \Lambda (\mathbf{x})\le k\},$$

where $k$ is chosen such that the test has a significance level $\alpha$.

This method is called the likelihood ratio test (LRT).

We can see that LRT is closely related to Wald Test with MLE: Wald statistic measures the closeness of the MLE to the null value (x-axis), while LRT measures the closeness of their likelihoods (y-axis). Under certain regularity conditions, the two measures are equivalent.

Link to original

CLT Test Statistic

Similar to CLT CI, CLT is also often used to construct a test statistic, and then the p-value, especially for HTs about the mean.

• ⭐️ Recall that a test statistic, or the HT itself, is to disprove null. Thus, a test statistic is often constructed as a function of ${\theta }_0$.

Suppose we want to test the null about mean $H_0:\theta ={\theta }_0$. Assuming null, CLT gives

$$\frac{\theta -{\theta }_0}{\mathrm{SE}(\theta )}\stackrel{d}{⟶}\mathcal{N}(0,1).$$

• ⭐️ Different from the Plug-in CI, we do not need to estimate the standard error using estimated $\theta$. Instead, we use the known ${\theta }_0$ to calculate the standard error.

More concretely, suppose the sample is $n$ iid Bernoulli r.v.s. Then the test statistic is

$$T_n=\sqrt{n}\frac{\overline{X}-{\theta }_0}{\sqrt{{\theta }_0(1-{\theta }_0)}}.$$

And the rejection region for a $\alpha$-level test is

$$\mathrm{RR}=\{\begin{array}{llc}{T}_n\ge z_{1-\alpha }& & \text{ (right-tail test)},\\ T_n\le z_{\alpha }& & \text{ (left-tail test)},\\ |T_n|\ge z_{1-\alpha /2}& & \text{ (two-sided test)},\end{array}$$

where $z_{\beta }$ is the $\beta$-quantile of the standard normal distribution.

Wald Test

In a more general parametric setting where we want to test some parameter $\theta$, we can construct an estimator and its asymptotic distribution using the estimated standard error:

$$W:=\frac{{\hat{\theta }}_X-{\theta }_0}{\widehat{\mathrm{SE}}({\hat{\theta }}_X)}\to \mathcal{N}(0,1),\text{under the null }{H}_0:{\theta }^{\ast }={\theta }_0.$$

The left-hand side $W$ is called the Wald test statistic. The rejection region is then $\{|W|>z_{1-\alpha /2}\}$ for a two-sided test, or $\{W\lessgtr z_{1-\alpha }\}$ for one-sided tests.

If the estimator if MLE, under sufficient regularity conditions, the SE is $\sqrt{(nI({\theta }_0){)}^{-1}}$. Similarly, the Wald test statistic is

$$W=\sqrt{nI({\hat{\theta }}_X^{\mathrm{MLE}})}\left({\hat{\theta }}_X^{\mathrm{MLE}}-{\theta }_0\right).$$

The use of estimated SE (variance) is also helpful for non-parametric tests. For example, if we want to test if two independent samples $X$ and $Y$ have the same mean, the Wald test statistic is

$$W=\frac{\overline{X}-\overline{Y}}{\widehat{\mathrm{SE}}(\overline{X}-\overline{Y})}=\frac{\overline{X}-\overline{Y}}{\sqrt{\widehat{\mathrm{Var}}(\overline{X})+\widehat{\mathrm{Var}}(\overline{Y})}}=\frac{\overline{X}-\overline{Y}}{\sqrt{\frac{{\hat{\sigma }}_X^2}{n}+\frac{{\hat{\sigma }}_Y^2}{m}}},$$

where $n$ and $m$ are the sample sizes of $X$ and $Y$, and we use sample means and sample variances as estimators.

Non-Asymptotic Test Statistic

Unlike CLT Test Statistic, which relies on CLT and Slutsky’s theorem for Wald Test to calculate the critical values, a non-asymptotic test statistic is more suitable for small sample sizes. A test statistic is non-asymptotic if we know its (approximate) distribution under the null without relying on asymptotic properties. If the underlying sample is already normal, then example distributions for non-asymptotic test statistics include Chi-Square Distribution and t Distribution. Other parts of the test procedure are the same as those using other test statistics.

p-Value

Introduction

p-value is the probability of obtaining a real-valued test statistic at least as extreme as the one actually obtained under the null hypothesis. In other words, (asymptotic) p-value of a test is the smallest (asymptotic) level $\alpha$ at which the test rejects $H_0$. Consider an observed test-statistic $t$ from unknown distribution $T$. Then the p-value $p$ is what the prior probability would be of observing a test-statistic value at least as “extreme” as $t$ if null hypothesis $H_0$ were true. That is:

• $p=\mathrm{Pr}(T\ge t|H_0)$ for a one-sided right-tail test,
• $p=\mathrm{Pr}(T\le t|H_0)$ for a one-sided left-tail test,
• $p=2\min \{\mathrm{Pr}(T\ge t\mid H_0),\mathrm{Pr}(T\le t\mid H_0)$ for a two-sided test.
  • If the distribution of $T$ is symmetric about zero, then $p=\mathrm{Pr}(|T|\ge |t|\mid H_0)$

CLT Test Statistic

$p=\{\begin{array}{ll}\mathrm{Pr}\left(T_n\ge t\right)& \text{(right-tail test)}\\ \mathrm{Pr}\left(T_n\le t\right)& \text{(left-tail test)}\\ \mathrm{Pr}\left(\left|T_n\right|\ge |t|\right)& \text{(two-sided test)}\end{array}$ where $t$ is the observed test statistic.

• ❗️ Since the test statistic is random, p-value is also random.

Fundamental rule of statistics

$\text{Reject }{\mathrm{H}}_0⟺p\text{-value}<\alpha$

In other words, an almost impossible event ($p<\alpha$) happens given H0, thus it is rejected.

• 💡 The smaller the p-value, the more confidently one can reject $H_0$, because the event is too unlikely to happen under null.

Link to original

Role of Alternative

Recall that

• Asymmetry in $H_0$ and $H_1$: the data is only used to try to disprove $H_0$. The result of an HT is either to reject or fail to reject the null hypothesis $H_0$.

Link to original

Also notice that in the calculation of the test statistic, critical value, and p-value, we only need the null hypothesis $H_0$. This brings up the question:

what is the role of the alternative hypothesis $H_1$?

We first remark that we do not expect the alternative to have the same critical role as the null, due to the asymmetry. However, the alternative do have two important implications:

• The alternative shapes the belief about the complement of the null. Specifically, the set pair $({\Theta }_0,{\Theta }_1)$ forms a model assumption, meaning that we believe the true parameter is either in ${\Theta }_0$ or ${\Theta }_1$. Under this belief, when rejecting $H_0$, we implicitly accept $H_1$.

  • 📗 For example, a company want to test their current risk control threshold ${\theta }_0$. Their hypotheses are $H_0:\theta ={\theta }_0$ and $H_1:\theta >{\theta }_0$. We can see that, they only reject the null if they believe the risk is higher then their current threshold. They do not modify the threshold (reject the null) even observing a risk significantly lower than the threshold, as it does no harm.

• The alternative dictates the direction of extremeness. When calculating the rejection region or p-value, it’s important to know what counts as an extreme event under the null. The alternative dictates the direction, i.e., right-tail, left-tail, or two-sided.

  • 📗 For example, suppose $H_0:\text{mean}={\mu }_0$ and $H_1:\text{mean}={\mu }_1$. Then, if ${\mu }_1>{\mu }_0$, it is extreme(ly unlikely the null is true) when we observe a large sample mean; on the other hand, if ${\mu }_1<{\mu }_0$, a small sample mean is extreme under the null.