Generalized Linear Model

A Generalized linear model (GLM) extends the classic Linear Regression model in two ways:

1. $Y\mid x\sim$ Exponential Family
2. Allows a link between the outcome and the predictors that satisfies $g(\mathbb{E}[Y\mid x])=x^T𝛽$
   • The function $g$ is called the link function
   • The regression function is $f_{\beta }(x)=(x\mapsto \mathbb{E}[Y\mid x])=g^{-1}(x^T\beta )$.

The generalized linear model can also be combined with Generalized Linear Regression, where the regression function is $x\mapsto \varphi (x{)}^T\beta$, which is still linear in the parameter, but $x$ could be transformed non-linearly into a new feature space. Together, we have the regression function $f_{\beta }(x)=g^{-1}(\varphi (x{)}^T\beta )$.

• 📗 If $\mathbb{E}[Y\mid x]=\frac{mx}{h+x}$, then $g(y)=y^{-1}$ and $\varphi (x)=x^{-1}$ satisfies the requirement, as $g(\mathbb{E}[Y\mid x])=\frac{1}{m}+\frac{h}{m}\frac{1}{x}=\varphi (1;x{)}^T(\frac{1}{m};\frac{h}{m})$.

Remarks

• Note that just like a Gaussian Linear Model, GLM is a model for the data-generating distribution that enable tractable Regression methods. In other words, it’s an assumption rather than a specific algorithm.
• We can use regression methods tailored for GLM even if the assumption does not hold. If we want to do so, we should design the best GLM that describes the data.
• Since the data-generating model is parametrized, the Regression task reduces to the Estimation of $\beta$, and MLE is a natural method for this task.
• Here the regression function is chosen to be the conditional mean. One can also extend the model to consider other regression functions.

Canonical GLM

Conventionally, GLM often refers to the canonical form that $Y\mid x$ follows a natural exponential dispersion model:

$$f\left(y_i;{\theta }_i,\varphi \right)=\exp \left(\frac{y_i{\theta }_i-b\left({\theta }_i\right)}{\varphi }+c\left(y_i,\varphi \right)\right),i=1,\dots ,n,$$

where $\varphi$ is called a dispersion parameter (and could be known); see more in Exponential Family. We remark that not every exponential family can be cast into this natural exponential dispersion model, but many common distributions can be.

Clearly, ${\theta }_i$ depends on $x_i$. In canonical GLM, we model the natural parameter linearly: $\theta =x^T\beta$, and link it to the conditional mean:

$$g(\mathbb{E}[Y\mid \theta ,\varphi ])=\theta =x^T\beta .$$

This is called the canonical link.

As calculated in Moments of Dispersion Exponential Family, we have $\mathbb{E}[Y\mid \theta ,\varphi ]=b^{\prime }(\theta )$. Thus, we have

$$g=b^{\prime -1}.$$

Furthermore, $\mathrm{Var}(Y\mid \theta ,\varphi )=\varphi b^{\prime \prime }(\theta )$. Therefore, if $\varphi >0$, we have

$$\mathrm{Var}(Y\mid \theta ,\varphi )>0⟹b^{\prime \prime }(\theta )>0⟹(g^{-1}{)}^{\prime }>0⟹g^{-1}\text{ is strictly increasing}⟹g\text{ is strictly increasing}.$$

A positive variance means the problem is not degenerated. And the strictly monotonicity also ensures the invertibility of $g$ and $b^{\prime }$, which further enables the identifiability of $\beta$.

MLE for GLM

We focus on MLE for canonical GLM. We use the nice properties of natural exponential dispersion family to derive the likelihood of $\beta$. We first have

$${\theta }_i=b^{\prime -1}(\mathbb{E}[Y_i\mid X_i])=b^{\prime -1}(g^{-1}(X_i^T\beta ))=(g\circ b^{\prime }{)}^{-1}(X_i^T\beta )=:h(X_i^T\beta ),$$

where $h=(g\circ b^{\prime }{)}^{-1}$. Then the log-likelihood is

$$\ell (\beta \mid (X,Y))=\sum _i\frac{Y_i{\theta }_i-b({\theta }_i)}{\varphi }+\text{Const.}=\sum _i\frac{Y_{i}h(X_i^T\beta )-b(h(X_i^T\beta ))}{\varphi }+\text{Const}.$$

If we use the canonical link, then $g=b^{\prime -1}$, and thus $h=\mathrm{id}$, i.e., ${\theta }_i=X_i^T\beta$; modeling ${\theta }_i$ by $X_i^T\beta$ is the motivation of the canonical link. Further, the MLE reduces to

$${\beta }^{\mathrm{MLE}}=\underset{\beta }{\mathrm{argmax}}\sum _i\frac{Y_i{X}_i^T\beta -b(X_i^T\beta )}{\varphi }.$$

Moreover, a positive $\varphi$, a positive variance (^bpp), and a full column rank $X=(X_1,\dots ,X_n{)}^T$ gives a negative definite Hessian of the log-likelihood:

$$\mathrm{Hess}(\ell (\beta ))=-{\varphi }^{-1}\sum _i\mathrm{Hess}(b(X_i^T\beta ))=-{\varphi }^{-1}\sum _i{b}^{\prime \prime }(X_i^T\beta )X_i{X}_i^T\prec 0.$$

Therefore, ${\beta }^{\mathrm{MLE}}$ is unique and can be attained by either Convex Optimization or solving the zero of the gradient:

$$\sum _i{Y}_i{X}_i-\sum _i{b}^{\prime }(X_i^T\beta )X_i=0.$$

• 💡 The asymptotic normality of MLE applies to GLMs.

Applications

Ordinary Least Squares by Gaussian Distribution

When the GLM is Gaussian Linear Model with canonical link, the MLE gives Ordinary Least Squares, the first regression method. In this case, $Y$ is a continuous numerical.

To see this, the natural exponential dispersion form of Gaussian has $b(\theta )={\theta }^2/2$ with a dispersion parameter ${\sigma }^2$. Then, $b^{\prime }(\theta )=\theta$ is linear, making the zero point of the log-likelihood (^zerog) satisfy

$$\sum _i{Y}_i{X}_i=\sum _i{X}_i{X}_i^T{\beta }^{\mathrm{MLE}}⟺X^{T}Y=X^{T}X{\beta }^{\mathrm{MLE}}⟺{\beta }^{\mathrm{MLE}}=(X^{T}X{)}^{-1}{X}^{T}Y.$$

Logistic Regression by Bernoulli Distribution

When the GLM is Bernoulli Distribution with canonical link, the MLE gives Logistic Regression. In this case, $Y$ is a categorical with two possible outcomes.

To see this, we have

$$f(y)=p^y(1-p{)}^{1-y}=\exp (y\ln (p/(1-p))+\ln (1-p))=:\exp (y\theta -\ln (1+e^{\theta }))=:\exp (y\theta -b(\theta )).$$

Therefore, the canonical link function is

$$g(p)=\ln \left(\frac{p}{1-p}\right).$$

This link is called the logit link, and $g^{-1}$ is called the sigmoid function:

$$p=b^{\prime }(\theta )=\frac{\exp (\theta )}{1+\exp (\theta )}=\frac{\exp (x^T\beta )}{1+\exp (x^T\beta )}.$$

Poisson Regression by Poisson Distribution

When the GLM is Poisson Distribution with canonical link, the MLE gives Poisson Regression. In this case, $Y$ is a count variable, which is a non-negative integer.

Other

• Multinomial distribution with MLE gives multinomial or ordinal regression, where $Y$ is categorical with more than two outcomes.
• Gamma Distribution with MLE gives Gamma regression, where $Y$ is positive, e.g., amount of an insurance claim.