Uniformly Most Powerful Test

The uniformly most powerful (UMP) test maximizes the power of the test for all values of the alternative hypothesis, given a fixed significance level. We start by defining the most powerful (MP) test for a simple alternative hypothesis ${\Theta }_1=\{{\theta }_1\}$. The MP test solves the following Constrained Optimization problem:

$$\begin{array}{rl}\underset{A}{\sup }& P_{{\theta }_1}(A(X)=1)\\ \text{s.t. }& \underset{{\theta }_0\in {\Theta }_0}{\sup }{P}_{{\theta }_0}(A(X)=1)\le \alpha .\end{array}$$

For a composite alternative hypothesis ${\Theta }_1$, we say a test is UMP, if it’s MP for all ${\theta }_1\in {\Theta }_1$. Formally, we consider the space of all randomized test $A:\mathcal{X}\to [0,1]$. Then, $A^{(\mathrm{UMP})}$ is UMP of size $\alpha$ if

$$\begin{array}{rlrl}{\mathbb{E}}_{{\theta }_1}[A^{(\mathrm{UMP})}(X)]=\underset{A:\mathcal{X}\to [0,1]}{\sup }& {\mathbb{E}}_{{\theta }_1}[A(X)]& & \text{(maximize power)}\\ \text{s.t. }& \underset{{\theta }_0\in {\Theta }_0}{\sup }{\mathbb{E}}_{{\theta }_0}[A(X)]\le \alpha & & \text{(size constraint)}\\ \forall {\theta }_1\in {\Theta }_1.& & & \text{(uniformity)}\end{array}$$

Neyman-Pearson

For a simple-simple HT, the Neyman-Pearson lemma states that the (U)MP test is a Likelihood Ratio Test given by

$$A^{(\mathrm{MP})}=A^{(\mathrm{NP})}(X)=1\left\{f_1(X)/f_0(X)>\lambda \right\},$$

where

$$P_{{\theta }_0}(f_1(X)/f_0(X)>\lambda )=\alpha .$$

Therefore, the (U)MP test for a simple-simple HT is also called the Neyman-Pearson optimal test.

• In general, if $A^{(\mathrm{MP})}=A^{(\mathrm{NP})}$ depends on ${\theta }_1$, it is not uniformly most powerful.

Monotone Likelihood Ratio

For certain Statistical Models, the NP optimal test evaluated at the boundary of ${\Theta }_0$ and ${\Theta }_1$ is UMP. We say a model ${\mathcal{P}}_{\Theta }$ has a monotone likelihood ratio if there exists a statistic $T(X)\in \mathbb{R}$ such that for any ${\theta }_0<{\theta }_1$, $P_{{\theta }_0}$ and $P_{{\theta }_1}$ are distinct and

$$\frac{f_{{\theta }_1}(X)}{f_{{\theta }_0}(X)}=\frac{g_1(T(X))}{g_0(T(X))},\text{for some }{g}_0,g_1\text{ with }{g}_1/g_0\text{ non-decreasing}.$$

For a statistical model with monotone likelihood ratio and a composite HT ${\Theta }_0=(-\infty ,{\theta }_0]$, ${\Theta }_1=({\theta }_0,\infty )$, we have the following results:

1. An UMP test exists and has the form

$$A^{(\mathrm{UMP})}(x)=\{\begin{array}{ll}1,& T(x)>c;\\ \gamma ,& T(x)=c;\\ 0,& T(x)<c,\end{array}$$

where $\gamma \in [0,1]$, and $c$, $\gamma$ are uniquely determined by the significance level constraint $\alpha$. 2. The power function $\beta (\theta )={\mathbb{E}}_{\theta }{A}^{\mathrm{UMP}}(X)$ is strictly increasing on $\{\theta :\beta (\theta )\in (0,1)\}$. 3. For any ${\theta }^{\prime }$, $A^{(\mathrm{UMP})}$ determines a test that is UMP for ${\Theta }_0^{\prime }=(-\infty ,{\theta }^{\prime }]$, ${\Theta }_1^{\prime }=({\theta }^{\prime },\infty )$ at level ${\alpha }^{\prime }=\beta ({\theta }^{\prime })$. 4. Among all size-$\alpha$ tests, $A^{(\mathrm{UMP})}$ minimizes ${\mathbb{E}}_{\theta }A(X)$ for $\theta \in (-\infty ,{\theta }_0)$.

• We note that $A^{(\mathrm{UMP})}$ is independent of ${\theta }_1$.

Proof

• This proof covers the Neyman-Pearson lemma.

(Simple). We first consider simple-simple HT for ${\theta }_0$ and an arbitrary ${\theta }_1\in {\Theta }_1$. Let $F_T(t)=P_0(T(X)\le t)$ be the CDF of $T(X)$ under ${\theta }_0$. For any $\alpha \in (0,1)$, let $c:=F_T^{-1}(1-\alpha )={\inf }_t\{F_T(t)\ge 1-\alpha \}$. Note that for any $c^{\prime }<c$, we have $F_T(c^{\prime })<1-\alpha$. Thus,

$$1-F_T(c)\le \alpha \le 1-F_T(c^{-}).$$

Let

$$A(x)=\{\begin{array}{l}1,T(x)>c;\\ \gamma ,T(x)=c;\\ 0,T(x)<c.\end{array}$$

We calculate the size of this test:

$${\mathbb{E}}_0[A(X)]=P_0(T(X)>c)+\gamma P_0(T(X)=c)=1-F_T(c)+\gamma (F_T(c)-F_T(c^{-})).$$

Therefore, we let $\gamma =\frac{\alpha -\alpha (c)}{\alpha (c^{-})-\alpha (c)}$, where $\alpha (c)=1-F_T(c)$, which is between $[0,1]$ as argued above.

(Sufficient). We then show that $A$ maximizes the power for ${\theta }_1$. For any other test $A^{\prime }$ such that ${\mathbb{E}}_0[A^{\prime }(X)]\le \alpha$, let $S^{+}:=\{A>A^{\prime }\}$ and $S^{-}:=\{A<A^{\prime }\}$. We see that on $S^{+}$, we must have $A(x)>0$ and thus $T(x)\ge c$; and on $S^{-}$, we must have $A(x)<1$ and thus $T(x)\le c$. Let $k=g_1(c)/g_0(c)$. Thus,

$$\begin{array}{rl}{\int }_{\mathcal{X}}(A(x)-A^{\prime }(x))(f_1(x)-k{f}_0(x))\mathrm{d}x=& {\int }_{\mathcal{X}}(A(x)-A^{\prime }(x))\left(1-\frac{g_0(T(x))}{g_1(T(x))}\frac{g_1(c)}{g_0(c)}\right)f_1(x)\mathrm{d}x\\ =& {\int }_{S^{+}}(\underset{>0}{\underbrace{A(x)-A^{\prime }(x)}})(1-\underset{\le 1}{\underbrace{\frac{g_0(T(x))}{g_1(T(x))}\frac{g_1(c)}{g_0(c)}}})f_1(x)\mathrm{d}x\\ & +{\int }_{S^{-}}(\underset{<0}{\underbrace{A(x)-A^{\prime }(x)}})(1-\underset{\ge 1}{\underbrace{\frac{g_0(T(x))}{g_1(T(x))}\frac{g_1(c)}{g_0(c)}}})f_1(x)\mathrm{d}x\\ \ge & 0.\end{array}$$

Therefore,

$${\mathbb{E}}_1[A(X)-A^{\prime }(X)]\ge k{\mathbb{E}}_0[A(x)-A^{\prime }(X)]=k\alpha -k{\mathbb{E}}_0[A^{\prime }(X)]\ge 0.$$

So $A$ is most powerful.

(Necessary). We also show that for any MP test $A^{\prime }$, $S^{\prime }:=\{A(x)\ne A^{\prime }(x)\}\cap \{f_1(x)\ne k{f}_0(x)\}$ is of zero measure. If not, similarly, we have

$${\int }_{\mathcal{X}}(A(x)-A^{\prime }(x))(f_1(x)-k{f}_0(x))\mathrm{d}x={\int }_{S^{\prime }}(A(x)-A^{\prime }(x))(f_1(x)-k{f}_0(x))\mathrm{d}x>0,$$

indicating that $A$ is more powerful, which is a contradiction. Further, if ${\mathbb{E}}_0[A^{\prime }(X)]<\alpha$ and ${\mathbb{E}}_1[A^{\prime }(X)]<1$, we can add points to the rejection region of $A^{\prime }$ to increase the power without violating the size constraint. Therefore, any MP test satisfies either ${\mathbb{E}}_0[A^{\prime }(X)]=\alpha$ or ${\mathbb{E}}_1[A^{\prime }(X)]=1$.

(Composite). First, since $A$ is independent of ${\theta }_1$, it is UMP for $(\{{\theta }_0\},{\Theta }_1)$. We then need to show Property (2), and the $A$ is UMP for $({\Theta }_0,{\Theta }_1)$. This is because no UMP $A^{\prime }$ for $({\Theta }_0,{\Theta }_1)$ would have a larger power than a UMP for $(\{{\theta }_0\},{\Theta }_1)$ (less constraint), and thus $A$ has the largest power for $({\Theta }_0,{\Theta }_1)$ as well (power is independent of ${\Theta }_0$).

(Property 2). For any $\theta$ such that $\beta (\theta )\in (0,1)$, we see that the above Neyman-Pearson test is UMP for $(\{\theta \},\{{\theta }^{\prime }\})$ with size $\beta (\theta )$ for any ${\theta }^{\prime }>\theta$. We now show that the power of the test $\beta ({\theta }^{\prime })$ is strictly larger than its size $\beta (\theta )$, hence proving the property. First, the constant test $A^{\prime }(x)\equiv \beta (\theta )$ has a size $\beta (\theta )$ and power $\beta (\theta )$. Since $A$ is UMP, it must have a power no less than $\beta (\theta )$. Now suppose $\beta ({\theta }^{\prime })=\beta (\theta )$. Then, the constant test $A^{\prime }$ is also UMP. By the (Necessary) condition, we have $T(x)\equiv c$, and thus $P_{\theta }\dot{=}{P}_{{\theta }^{\prime }}$, contradicting the assumption that two distributions are distinct. Therefore, $\beta ({\theta }^{\prime })>\beta (\theta )$.

(Property 3). Note that $c$ depends on $\theta$ and $\alpha$, and for a fixed $c$, the same test $A$ if UMP for $(-\infty ,{\theta }^{\prime }],({\theta }^{\prime },\infty )$ with size $\beta ({\theta }^{\prime })$.

(Property 4). One obtains a test that minimizes the power by flipping the rejection region of the UMP test. Therefore, $A$ maximizes the power for $\theta \in ({\theta }_0,\infty )$ and minimizes the power for $\theta \in (-\infty ,{\theta }_0)$.

Exponential Family

An important class of models with monotone likelihood ratio is the Exponential Family, which includes many common distributions such as the Normal Distribution, Poisson Distribution, and Exponential Distribution. Recall that a one-parameter exponential family has the form

f(x ; \theta)=c(\theta) h(x) \exp (t(x) q(\theta))

Link to original

As a corollary of the above results, if $q(\theta )$ is strictly monotone, then this model admits an UMP test for ${\Theta }_0=(-\infty ,{\theta }_0]$, ${\Theta }_1=({\theta }_0,\infty )$. Specifically, if $q$ is monotonically increasing, then the UMP test has the same form as $A^{(\mathrm{UMP})}$; if $q$ is monotonically decreasing, then the UMP test has the form of $A^{(\mathrm{UMP})}$ with reversed inequalities.

Existence of UMP implies exponential family\

We state that an UMP exists for exponential family models. Surprisingly, the other direction is also generally true. Under weak conditions, the exists of UMP for one-sided composite HT of level $\alpha$ and all sample sizes implies an exponential family model.

Gaussian

We consider ${\mathcal{P}}_{\theta }=\mathcal{N}(\theta ,1)$ as a concrete example. Suppose ${\Theta }_0=(-\infty ,0]$, ${\Theta }_1=(0,\infty )$. Then,

$$A_{\alpha }^{(\mathrm{UMP})}(x)=\mathrm{𝟙}\{x>{\Phi }^{-1}(1-\alpha )\}.$$

To prove this, we first consider a simple-simple HT ${\Theta }_0=\{0\}$, ${\Theta }_1=\{{\theta }_1\},{\theta }_1>0$. By the Neyman-Pearson lemma, the optimal test for this simple-simple HT is

$$A_{\alpha }^{(\mathrm{NP})}(x)=\mathrm{𝟙}\{f_1(x)/f_0(x)>\lambda \},$$

where $\lambda$ is determined by $P_0(f_1(X)/f_0(X)>\lambda )=\alpha$.

Note that for this simple-simple HT,

$$f_1(x)/f_0(x)=\exp \left(-\frac{1}{2}(x-{\theta }_1{)}^2+\frac{1}{2}{x}^2\right)=\exp \left({\theta }_{1}x-\frac{1}{2}{\theta }_1^2\right),$$

which monotonically increases in $x$. Thus, there exists $\eta$ such that

$$P_0(f_1(X)/f_0(X)>\lambda )=\alpha ⟺P_0(x>\eta )=\alpha ⟺\eta ={\Phi }^{-1}(1-\alpha ).$$

$A_{\alpha }^{(\mathrm{NP})}$ is independent of ${\theta }_1$, and is thus UMP for the simple-composite HT ${\Theta }_0=\{0\}$, ${\Theta }_1=(0,\infty )$.

On the other hand, for any ${\theta }_0<0$, we have

$${\mathbb{E}}_{{\theta }_0}{A}^{(\mathrm{NP})}(X)=P_{{\theta }_0}(X>{\Phi }^{-1}(1-\alpha ))<P_0(X>{\Phi }^{-1}(1-\alpha ))=\alpha .$$

Thus, $A_{\alpha }^{(\mathrm{NP})}$ satisfies the size constraint for the composite null ${\Theta }_0=(-\infty ,0]$. In conclusion, $A^{(\mathrm{UMP})}=A^{(\mathrm{NP})}$.