The renormalization group and information bottleneck: a unified framework

Achieving useful simplified descriptions of high-dimensional systems is a fundamental problem in statistical physics. A central issue is formalizing which details should be retained and which discarded, such that we are left with only “relevant” information. In statistical physics, the outcome of the renormalization group is a reduced description where we are left with an accurate description of the macroscopic behavior of the system. In information theory, we use the information bottleneck to determine the optimal balance between features we accurately convey and those that are irrelevant complexity. Here we present an approach that unifies the concepts of the renormalization group and the information bottleneck. We achieve a coarse-graining procedure where we can control what “relevant” information we choose to keep, e.g. retaining information about long-distance features while removing local information. Studying the method in the information plane allows us to automatically select the best representation at each size. Variational approaches allow us to scale up our implementation, so that this approach can be successfully applied to large systems. We test our method on a variety of datasets from both physics and machine learning.