Conference paper
Difference-of-Convex optimization for variational kl-corrected inference in dirichlet process mixtures
Variational methods for approximate inference in Bayesian models optimise a lower bound on the marginal likelihood, but the optimization problem often suffers from being nonconvex and high-dimensional. This can be alleviated by working in a collapsed domain where a part of the parameter space is marginalized.
We consider the KL-corrected collapsed variational bound and apply it to Dirichlet process mixture models, allowing us to reduce the optimization space considerably. We find that the variational bound exhibits consistent and exploitable structure, allowing the application of difference-of-convex optimization algorithms.
We show how this yields an interpretable fixed-point update algorithm in the collapsed setting for the Dirichlet process mixture model. We connect this update formula to classical coordinate ascent updates, illustrating that the proposed improvement surprisingly reduces to the traditional scheme.
Language: | English |
---|---|
Publisher: | IEEE |
Year: | 2017 |
Pages: | 1-6 |
Proceedings: | 27th International Workshop on Machine Learning for Signal Processing |
ISBN: | 1509063412 , 1509063420 , 9781509063413 and 9781509063420 |
Types: | Conference paper |
DOI: | 10.1109/MLSP.2017.8168159 |
ORCIDs: | Schmidt, Mikkel Nørgaard and Mørup, Morten |
Bayesian nonparametrics Collapsed methods Difference-of-convex optimization Variational inference
Bayes methods Bayesian models Convex functions Dirichlet process mixture model Entropy Gaussian processes Mathematical model Mixture models Optimization Standards approximate inference collapsed domain collapsed methods collapsed setting convex programming coordinate ascent updates difference-of-convex optimization difference-of-convex optimization algorithms exploitable structure inference mechanisms interpretable fixed-point update algorithm marginal likelihood optimization problem optimization space parameter space variational bound exhibits variational inference variational kl-corrected inference variational methods variational techniques