Estimating Mutual Information with the Deep Variational Symmetric Information Bottleneck

Mutual Information (MI) captures nonlinear statistical relations between two variables. MI has proved to be useful in analysis of complex systems, in methods involving clustering, feature selection, and dimensionality reduction, among others. Estimation of MI between high-dimensional variables is a challenging task, often requiring impractically large sample sizes for accurate estimation, and thus limiting wider adoption of methods. One approach to resolve the sampling issue is to reduce the dimensionality of the variables. However, such a reduction can destroy correlations between the variables. We resolve this problem using the Deep Variational Symmetric Information Bottleneck (DVSIB), which simultaneously compresses the variables X and Y into two corresponding lower dimensional latent variables Z_X and Z_Y, while maximizing the information between the latent variables. The information between Z_X and Z_Y produced by DVSIB can be used as a proxy for the information between X and Y. We demonstrate the effectiveness of this method by assessing its performance on synthetic and real datasets, showcasing its robustness and accuracy. We show that our method can estimate mutual information between two high-dimensional variables in many cases where standard estimators fail.