Maximum a Posteriori

As in Bayesian Inference, we have a posterior distribution given a prior distribution and the observation. However, instead of taking the expectation, we can also apply the Maximum Likelihood Estimation, which gives the maximum a posteriori (MAP) estimation.

$$\begin{array}{rl}{w}_{\mathrm{MAP}}& =\underset{w}{\mathrm{argmax}}\ln p(w\mid y,X)\\ & =\underset{w}{\mathrm{argmax}}\ln p(y\mid w,X)+\ln p(w)-\ln p(y\mid X)\\ & =\underset{w}{\mathrm{argmax}}\ln p(y\mid w,X)+\ln p(w)\end{array}$$

Note here, we are not maximizing the likelihood of the observation; we are maximizing the likelihood of the weight given the posterior distribution.

Relationship with Ridge Regression

Just like MLE being the probability interpretation of least square estimation, MAP is the probability interpretation of the Ridge Regression.

Assume the prior distribution of $w$ is $\mathcal{N}(0,{\lambda }^{-1}I)$, then

$$w_{\mathrm{MAP}}=\arg \underset{w}{\max }-\frac{1}{2{\sigma }^2}(y-Xw{)}^T(y-Xw)-\frac{\lambda }{2}{w}^{T}w$$

which gives $w_{\mathrm{MAP}}=(\lambda {\sigma }^{2}I+X^{T}X{)}^{-1}{X}^{T}y$, equaling to $w_{\mathrm{RR}}$ with regularizer parameter $\lambda {\sigma }^2$.