Bayesian Logistic Regression

As stated in Solution, giving the weight $w$ a normal prior distribution can ameliorate its overfitting phenomena. But unlike Ridge Regression/Bayesian Linear Regression, where we can get a closed solution/posterior distribution, the Logistic Sigmoid makes it impossible to calculate the posterior of $w$ in Logistic Regression.

One solution is to approximate the posterior using a parametrized distribution, e.g., Normal Distribution, and then approximate the parameter. Notice that the posterior has the expression

$$p(w|x,y)=\frac{p(w,y|x)}{p(y|x)}=\frac{\exp (\ln p(w,y|x))}{\int \exp (\ln p(w,y|x))\mathrm{d}w}.$$

Then, by Taylor expansion, we can make $\ln p(w,y|x)$ a quadratic term, which gives a Normal Distribution approximating the posterior. This is called the Laplace approximation. Let $f(w)=p(w,y|x)$. Expanding $\ln f(w)$ to the second order at MAP gives

$$\exp (\ln f(w))\approx \exp \left(\ln f(w_{\mathrm{MAP}})+0+\frac{1}{2}(w-w_{\mathrm{MAP}}{)}^T{\nabla }^2\ln f(w_{\mathrm{MAP}})(w-w_{\mathrm{MAP}})\right),$$

where $0$ is given by the definition of $w_{\mathrm{MAP}}$. And $w$-independent constants are canceled out from both the denominator and the numerator. Therefore, we get $p(w|x,y)\approx N(\mu ,\Sigma )$, where

$$\mu =w_{\mathrm{MAP}},\Sigma =(-{\nabla }^2\ln p(w_{\mathrm{MAP}},y|x){)}^{-1}.$$

Furthermore, by taking the prior $w\sim \mathcal{N}(0,{\lambda }^{-1}I)$, we have

$$w_{\mathrm{MAP}}=\underset{w}{\mathrm{argmax}}\sum \ln \sigma (y_i{x}_i^{T}w)-\frac{\lambda }{2}{w}^{T}w,\text{ no closed solution}$$

and by Property 2 of Logistic Sigmoid, we get

$${\nabla }^2\ln p(y,w|x)={\nabla }^2\left(\sum \ln \sigma (y_i{x}_i^{T}w)+C-\frac{\lambda }{2}{w}^{T}w\right)=-\lambda I-\sum {\sigma }_i(1-{\sigma }_i)x_i{x}_i^T.$$