Variational Inference with Heavy-Tailed Distributions using the Coupled Free Energy

The geometry of the coupled exponential family generalizes the exponential family, providing a simple separation between nonlinear and linear sources of uncertainty. In this framework, the nonlinear statistical coupling κ measures the shape, nonlinearity, and complexity; the independent equals or Tsallis parameter q measures the number of independent random variables in the same state and is dependent on the coupling, the location shape, and the dimensions. The generalized moments of the coupled exponential distributions, which are a function of the independent equals distribution, are guaranteed to be finite. From this foundation, we derive the information geometric definitions for the coupled entropy and its maximizing distributions, the coupled exponential family.

By clearly defining the physical properties of linear uncertainty (scale) and nonlinear uncertainty (shape/coupling/complexity), the coupled entropy resolves shortcomings in the Rényi and Tsallis entropy functions. Together, the coupled exponential family and coupled entropy function provide an analytical framework for modeling complex systems across diverse domains, including physics, finance, climate, and artificial intelligence, which exhibit heavy-tailed phenomena and nonlinear interdependencies. The generalized Fisher metric, its gradient, and the dual connectors are derived, providing the necessary tools for designing innovative machine learning methods. We introduce an optimization framework for variational inference based on the coupled free energy, extending variational inference techniques to account for the curved geometry of the coupled exponential family. By leveraging the coupled free energy, which is equal to the coupled evidence lower bound (ELBO) of the inverted probabilities, we improve the accuracy and robustness of the learned model. The method is applied to the design of a coupled variational autoencoder (CVAE). The novelty stems from sampling the heavy-tailed latent distribution with its associated coupled probability, which has faster-decaying tails. The result is the ability to train a model that is robust against severe outliers, while ensuring that the training process remains stable.