Testing for Regression

Regression studies the effect of the explanatory variable $X$ on the response variable $Y$. After solving the regression task, we want to know if specific features of $X$ have significant effects on $Y$. We use Hypothesis Testing for this purpose.

For example, in a linear regression model:

$$Y=f_{\theta }(X)+ϵ=X\theta +ϵ,$$

we want to test

$$H_0:{\theta }_j=0\text{vs}{H}_1:{\theta }_j\ne 0.$$

Other regression models and hypotheses can be considered. We can also test a group of features using Multiple Hypothesis Testing.

Gaussian Linear Model

We focus the above example with a Gaussian Linear Model, i.e., $ϵ\sim \mathcal{N}(0,{\sigma }^{2}I)$. We have two useful lemmas:¹

1. Let ${\hat{\sigma }}^2:=\Vert Y-X{\hat{\theta }}^{\mathrm{LS}}{\Vert }_2^2/(n-d)$, where $d$ is feature dimension ($X\in {\mathbb{R}}^{n\times d}$). Then $(n-d){\hat{\sigma }}^2/{\sigma }^2\sim {\chi }_{n-d}^2$.
2. ${\hat{\theta }}^{\mathrm{LS}}\perp \perp {\hat{\sigma }}^2$.

Combining the two lemmas gives the test statistic:

$$T_n^{(j)}:=\frac{\frac{{\hat{\theta }}_j^{\mathrm{LS}}-{\theta }_j}{\sqrt{{\sigma }^2(X^{T}X{)}_{jj}^{-1}}}}{\sqrt{\frac{{\hat{\sigma }}^2}{{\sigma }^2}}}=\frac{{\hat{\theta }}_j^{\mathrm{LS}}-0}{\sqrt{{\hat{\sigma }}^2(X^{T}X{)}_{jj}^{-1}}}\sim t_{n-d},$$

where $A_{jj}$ is the $j$-th diagonal element of the matrix $A$.

Footnotes

1. Both lemmas can be proved using the derivation in Minimax Estimator or ^ls-pred. Specifically, for the first item, $Y-X{\hat{\theta }}^{\mathrm{LS}}={\mathrm{Proj}}_{\mathrm{col}(X{)}^{\perp }}ϵ\sim \mathcal{N}(0,{\sigma }^2{I}_{n-d})$ and thus $(n-d){\hat{\sigma }}^2/{\sigma }^2=\Vert Y-X{\hat{\theta }}^{\mathrm{LS}}{\Vert }^2/{\sigma }^2\sim {\chi }_{n-d}^2$. For the second item ${\mathrm{Proj}}_{\mathrm{col}(X)}Y$ is sufficient for $\theta$ and perpendicular to ${\mathrm{Proj}}_{\mathrm{col}(X{)}^{\perp }}ϵ$, and thus $\theta \perp \perp {\hat{\sigma }}^2$. ↩