Confidence Interval and Hypothesis Test Duality

Beyond both being statistical decision-making procedures, confidence intervals and hypothesis tests have a stronger connection. The one direction of the duality is easy: given a confidence interval $C(X)$ for the parameter, we can construct a hypothesis test $\psi (X)=1\{{\theta }_0\notin C(X)\}$. Essentially, the confidence region corresponds to the rejection region of the hypothesis test.

For the other direction, we consider a family of tests $\{{\psi }_{{\theta }_0}{\}}_{{\theta }_0\in \Theta }$, each of which rejects the null hypothesis $H_0:\theta ={\theta }_0$ with probability $\alpha$. That is, the family of tests test against all possible values of the parameter $\theta$. Then, we can construct a confidence interval

$$C(X)=\{\theta :{\psi }_{\theta }(X)=0\}.$$

Recall the definition of the level of a Hypothesis Testing. We have

$$P(C(X)\not\ni \theta )=P_{\theta }({\psi }_{\theta }(X)=1)\le \alpha ,$$

indicating that $C(X)$ is indeed a $(1-\alpha )$ level confidence interval. In other words, to build a $(1-\alpha )$ level CI, we collect all parameters that are consistent enough with the data, i.e., those that do not reject of the simple null hypothesis at level $\alpha$.

Algorithm for HT to CI

We give an example algorithm for constructing CI through HT. Our goal is to construct a CI with balanced coverage: $P(\theta >\sup C(X))\approx P(\theta <\inf C(X))$.

Algorithm

• Input: level $\alpha$
• For $\tilde{\theta }\in \Theta$:
  • Generate a $\alpha /2$-test ${\psi }_{\tilde{\theta }}^{\uparrow }$ on HT ${\Theta }_0=\{\tilde{\theta }\}$ against ${\Theta }_1=\{\theta >\tilde{\theta }\}$.
  • Generate a $\alpha /2$-test ${\psi }_{\tilde{\theta }}^{\downarrow }$ on HT ${\Theta }_0=\{\tilde{\theta }\}$ against ${\Theta }_1=\{\theta <\tilde{\theta }\}$.
• Return: $C(X)=\{\tilde{\theta }\in \Theta :{\psi }_{\tilde{\theta }}^{\uparrow }(X)={\psi }_{\tilde{\theta }}^{\downarrow }(X)=0\}$

One can check that the above method returns a finite-sample valid CI.

HT CI for a Few Bernoulli Trials

We construct a Confidence Interval for the parameter $p$ given a few Bernoulli Trials to demonstrate how HT-based CI adapts to the number of samples and is finite-sample valid.

First, we note that when we only have a few samples, it’s more likely we observe extreme events like $\overline{X}=0$ or $\overline{X}=1$. In such cases, non-inclusive Wald CI gives an degenerated CI:

$$C^{(\mathrm{Wald})}(X)=\hat{p}\pm z_{1-\alpha /2}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}=\{\hat{p}\},$$

which is obviously not valid.

We now examine the HT-based CI constructed by Algorithm for HT to CI under such extreme events. Consider $n$ trials from $\mathrm{Binom}(n,p)$. Since the Binomial Distribution has a monotone likelihood function, we know the UMP level-$\alpha$ test is

$${\psi }_{\tilde{\theta }}^{\uparrow }(X)=1\{X>q_{1-\alpha /2}^{\tilde{\theta }}\},{\psi }_{\tilde{\theta }}^{\downarrow }(X)=1\{X<q_{\alpha /2}^{\tilde{\theta }}\},$$

where $q_{\beta }^{\tilde{\theta }}$ is the $\beta$-quantile of $\mathrm{Binom}(n,\tilde{\theta })$. Now suppose $\hat{p}=\overline{X}=0$. We need to find the parameter $\tilde{\theta }$ such that

$${\psi }_{\tilde{\theta }}(X)={\psi }_{\tilde{\theta }}^{\uparrow }(X)\vee {\psi }_{\tilde{\theta }}^{\downarrow }(X)={\psi }_{\tilde{\theta }}^{\downarrow }(X)=1\{q_{\alpha /2}^{\tilde{\theta }}>0\},$$

where the second equality uses the fact that $1\{0>q_{1-\alpha /2}^{\tilde{\theta }}\}\equiv 0$. Although the general binomial quantile function is hard to express, in such extreme events, we have

$$\begin{array}{rl}{\psi }_{\tilde{\theta }}(X)=0⟺& 1\{q_{\alpha /2}^{\tilde{\theta }}>0\}=0\\ ⟺& q_{\alpha /2}^{\tilde{\theta }}=0\\ ⟺& P_{\tilde{\theta }}(X\le 0)\ge \alpha /2\\ ⟺& n(1-\theta {)}^n\ge \alpha /2\\ ⟺& \tilde{\theta }\le 1-{\left(\frac{\alpha }{2n}\right)}^{1/n}.\end{array}$$

Therefore, we get the CI

$$C^{(\mathrm{HT})}(X)=\left[0,1-{\left(\frac{\alpha }{2n}\right)}^{1/n}\right],$$

which is not degenerated and is valid.