Local Learning Rules for Out-of-Equilibrium Physical Generative Models

Generative models have broad societal applications ranging from image generation to protein discovery. Among them, diffusion-based models are the state of the art for generative modeling, yet they are extremely energy-intensive: generating a single image can consume as much energy as half a smartphone charge. Embedding such models in physical systems—where stochastic dynamics and thermal noise are naturally present—offers a path toward energy-efficient generative AI.

We show that the out-of-equilibrium driving protocol of score-based generative models (SGMs) can be learned through local learning rules. The gradient with respect to the parameters of the driving protocol is obtained directly from measured forces or from observed system dynamics, without any digital simulations. This enables on-device training. As a numerical demonstration, we implement an SGM in a network of driven, nonlinear, overdamped oscillators coupled to a thermal bath. The model successfully learns to sample from a two-dimensional mixture of Gaussians and, when scaled to a 12 × 12 oscillator network, generates images of handwritten digits “0” and “1” from the MNIST dataset.