Temperature-tuning trained energy functions improves generative performance

When models are fit to real data of complex multi-dimensional systems, ideally one expects the learned distribution to be able to generate states exemplary of the ground truth. In practice, however, this is often not the case; specialized sampling must be performed in order to generate desired outputs, especially if the training objective does not align well with desired performance. Here we utilize minimal and physically motivated energy-based models in order to systematically interrogate the differences between the data-generation processes of ground truth and learned models sampled at varying temperatures. Energy-based modeling is commonly used for learning generative models in machine learning and inspired by the Boltzmann distribution from statistical physics. This lends itself well to an examination of the surprising ability of temperature tuning of learned energy functions—a simple one parameter re-scaling used in practice but not well understood—to improve sampling performance. Whether the post-hoc sampling temperature need be raised or lowered, and by how much, depends on several factors: choice of objective function, fitting protocol, amount of samples, and most importantly, properties of order and disorder inherent to the true system. This implies that the post-hoc tuning often done in practice is revelatory of properties not explicitly learned by the fit models themselves.