The Bayesian Lasso

Trevor Park, George Casella – 2008

Abstract

The Lasso estimate for linear regression parameters can be interpreted as a Bayesian posterior mode estimate when the regression parameters have independent Laplace (i.e., double-exponential) priors. Gibbs sampling from this posterior is possible using an expanded hierarchy with conjugate normal priors for the regression parameters and independent exponential priors on their variances. A connection with the inverse Gaussian distribution provides tractable full conditional distributions. The Bayesian Lasso provides interval estimates (Bayesian credible intervals) that can guide variable selection. Moreover, the structure of the hierarchical model provides both Bayesian and likelihood methods for selecting the Lasso parameter. Slight modifications lead to Bayesian versions of other Lasso-related estimation methods, including bridge regression and a robust variant.

Summary

TK

The nice part about a fully Bayesian approach to Lasso is that you don’t have to worry about doing cross-validation. You just set your priors and you’re ready to go. Even the $\lambda$ parameter in the Laplace coefficient prior can be set via maximum likelihood.

Hierarchical priors

• Prior on coefficients, $\beta$
• Prior on variance of coefficients, ${\sigma }^2$
• Prior on the Laplace parameter, $\lambda$

Priors

Prior on coefficients, $\beta$ (conditional Laplace distribution): $p(\beta |{\sigma }^2)={\prod }_{j=1}^p\frac{\lambda }{2\sqrt{{\sigma }^2}}{e}^{-\lambda |{\beta }_j|/\sqrt{{\sigma }^2}}$ $\lambda$ here is equivalent to $\sqrt{{\sigma }^2}/b$, where b is the scale parameter of the Laplace distribution. So in a way $\lambda$ is the inverse variance / precision.

Prior on variance of coefficients, ${\sigma }^2$ (non-informative scale-invariant): $p({\sigma }^2)=1/{\sigma }^2$

Note that it’s presumed the data has been standardized, so the above is technically scale-invariant. Also note that this kind of prior is very similar to an exponential, one-sided Laplace, or Gamma distribution with shape parameter = 1.

Prior on the Laplace parameter, $\lambda$ (note ${\lambda }^2$, not $\lambda$): $p({\lambda }^2)=\frac{{\delta }^2}{\Gamma (r)}({\lambda }^2{)}^{r-1}{e}^{-\delta {\lambda }^2}$ with $r=1$ for most use cases and $\delta >1$. This is a Gamma distribution. If $r=1$ it actually becomes an exponential distribution.

Estimation

The authors propose a Gibbs sampling approach to estimating the Bayesian Lasso, which exploits the fact that the Laplace distribution can be represented as a mixture of normal distributions.

---

References

paperreadonline