Statistical Mechanics of Reinforcement Learning

Reinforcement learning, the study of optimal decision-making over long timescales in stochastic systems, has recently seen remarkable advances due in large part to the efforts of the deep learning community. However, a theoretical framework to understand and develop the corresponding algorithms is lacking. We show that the reinforcement learning problem can be formulated and solved using statistical mechanics. Drawing on principles of invariance and duality, and using non-equilibrium tools such as large deviation theory, important problems in the field, such as inverse RL, reward shaping, and bounding the value function, can be addressed. In this talk, we describe this framework, showcasing its ability to solve open problems in average-reward reinforcement learning. Based on its broad applicability, we also discuss implications for RL applications in complex high-dimensional physical systems.