Deep Variational Multivariate Information Bottleneck

Variational dimensionality reduction methods are known for their high accuracy, generative abilities, and robustness. We introduce a unifying principle rooted in information theory to rederive, generalize, and design variational methods. We base our framework on an interpretation of the multivariate information bottleneck, in which the information in an encoder graph is traded off against the information in a decoder graph. The encoder graph specifies the compression of the data and the decoder graph specifies a generative model for the data. Using this framework, we can rederive the deep variational information bottleneck and variational autoencoders, and we generalize deep variational CCA (DVCCA) to beta-DVCCA. We also design a new method, deep variational symmetric informational bottleneck (DVSIB), which simultaneously compresses two variables to preserve information between their compressed representations. We implement all these algorithms and evaluate their ability to produce shared low dimensional latent spaces on a modified noisy MNIST dataset. We show that algorithms that are better matched to the structure of the data (beta-DVCCA and DVSIB in our case) produce better latent spaces as measured by classification accuracy and the dimensionality of the latent variables.