Order Statistics

Given a sample $X=(X_1,\dots ,X_n)$ with components being iid Random Variables, the order statistics of $X$ is a Statistic, i.e., the sorted values of the sample: $X_{(1)}\le X_{(2)}\le \cdots \le X_{(n)}$. Order statistics are Sufficient Statistic.

• Order statistics are usually considered for continuous random variables, for which samples will have distinct values with probability 1.

Maximum and Minimum

The distributions of $X_{(1)}={\min }_i{X}_i$ and $X_{(n)}={\max }_i{X}_i$ are easy to compute. The CDFs are

$$\begin{array}{rl}P(X_{(n)}\le x)=& P(X_1,\dots ,X_n\le x)=F(x{)}^n,\\ P(X_{(1)}\le x)=& 1-P(X_{(1)}>x)=1-P(X_1,\dots ,X_n>x)=1-(1-F(x){)}^n.\end{array}$$

If $X_i$ is a continuous r.v. with Density $f$, we have

$$f_{X_{(n)}}(x)=n{F}^{n-1}(x)f(x),f_{X_{(1)}}(x)=n(1-F(x){)}^{n-1}f(x).$$

For Uniform Distribution, these correspond to Beta Distribution: $X_{(n)}\sim \mathrm{Beta}(n,1)$ and $X_{(1)}\sim \mathrm{Beta}(1,n)$.

We plot the PDF of $X_{(n)}$, $X_{(1)}$, and $X_i$ below

f <- function(x) {
  ifelse(x < 0 | x > 1, 0, 1)
}
 
f_min <- function(x, n) {
  n * (1 - x)^(n - 1) * f(x)
}
 
f_max <- function(x, n) {
  n * x^(n - 1) * f(x)
}
 
x <- seq(0, 1, length.out = 1000)
plot(x, f_max(x, 10), type = "l", col = "blue", xlim=c(0,1), xlab = "x", ylab = "Density", main = "PDF of Order Statistics with n=10")
lines(x, f_min(x, 10), type = "l", col = "red")
lines(x, f(x), col = "black")
legend("topright", legend = c("Max", "Min", "i-th"), col = c("blue", "red", "black"), lty = 1)

Joint Distribution

Wrong

One tempting way to compute the joint distribution of order statistics is to use Conditional Probability, leading to the plausible formula:

$$\begin{array}{rl}{f}_{(X_{(i)})}(x)=& \sum _{\sigma \in S_n}{f}_{(X_{(i)})\mid X_{\sigma }}(x\mid X_{(i)}=X_{\sigma (i)})P(X_{(i)}=X_{\sigma (i)})\\ =& \sum _{\sigma \in S_n}{f}_{X_{\sigma }}(x)P(X_{(i)}=X_{\sigma (i)})\\ =& n!\prod _{i=1}^{n}f(x_i)\cdot 1/n!\\ =& \prod _{i=1}^{n}f(x_i).\end{array}$$

where $S_n$ is the set of all permutations of $n$ elements. The first, third, and fourth equalities are correct. The second equality seems correct, as given that $X_{(i)}=X_{\sigma (i)}$, the density of $f_{(X_{(i)})}$ should be the same as $f_{X_{\sigma }}$. However,

$$P(X_{\sigma }\in B\mid X_{\sigma (i)}=X_{(i)})\ne P(X_{\sigma }\in B).$$

An intuitive explanation is that when $X_{\sigma (1)}=X_{(1)}$, then the distribution of $X_{\sigma (1)}$ is no longer uniform, as the other $X_{\sigma (i)}$ are constrained to be larger than $X_{(1)}$. Therefore, $f_{X_{(i)}\mid X_{\sigma (i)}=X_{(i)}}(x\mid X_{\sigma (i)}=X_{(i)})\ne f_{X_{\sigma }}(x)$.

We first work with probabilities, which are easier to work with than PDFs. For values $a_1<b_1\le a_2<b_2\le \dots a_n<b_n$, we have

$$\left\{(X_{(i)})\in {\text{bigtimes}}_i(a_i,b_i)\right\}=\left\{X_1\in (a_1,b_1),X_2\in (a_2,b_2)\dots \right\}\cup \left\{X_2\in (a_1,b_1),X_1\in (a_2,b_2)\right\}\cup \cdots =\underset{\sigma \in S_n}{⨆}\left\{(X_{\sigma (i)})\in {\text{bigtimes}}_i(a_i,b_i)\right\}.$$

Therefore,

$$\begin{array}{rl}P(a_1<X_{(1)}\le b_1,\dots ,a_n<X_{(n)}\le b_n)=& \sum _{\sigma \in S_n}P(a_1<X_{\sigma (1)}<b_1,\dots ,a_n<X_{\sigma (n)}<b_n)\\ =& \sum _{\sigma \in S_n}{\int }_{a_1}^{b_1}\cdots {\int }_{a_n}^{b_n}{f}_{X_{\sigma }}(x_1,\dots ,x_n)\mathrm{d}{x}_1\dots \mathrm{d}{x}_n\\ =& \sum _{\sigma \in S_n}{\int }_{a_1}^{b_1}f(x_1)\mathrm{d}{x}_1\cdots {\int }_{a_n}^{b_n}f(x_n)\mathrm{d}{x}_n\\ =& n!\prod _i(F(b_i)-F(a_i)).\end{array}$$

Taking the derivative gives

$$f_{(X_{(i)})}(x)=n!\prod _{i}f(x_i)1\{x_1\le \cdots \le x_n\}.$$

Distribution of i-th Order Statistic

By taking the marginal distribution of $X_{(i)}$, we have

$$f_{X_{(i)}}(x_i)=\frac{n!}{(n-i)!(i-1)!}{F}^{i-1}(x_i)(1-F(x_i){)}^{n-i}f(x_i).$$