Mode-Assisted Unsupervised Learning of Restricted Boltzmann Machines

Restricted Boltzmann Machines (RBMs) are a powerful class of unsupervised models well known in machine learning. However, unlike their more popular supervised counterparts, their training requires computing a gradient that is notoriously difficult even to approximate. In this work we show that properly combining standard gradient approximations with an off-gradient direction, constructed from samples of the RBM ground state (mode), improves their training dramatically over the standard methods. This approach, which we call "mode training", promotes faster training and stability, in addition to lowering the converged relative entropy (KL divergence). We report promising preliminary results with small models on synthetic data sets and discuss extensions to more realistic scenarios, where a physics-based approach, memcomputing [1], is used
to sample the mode efficiently. The mode training we suggest is versatile, as it can be applied with any given gradient method, and is easily extended to more general energy-based neural network structures such as deep, convolutional and unrestricted Boltzmann machines. [1] M. Di Ventra and F.L. Traversa, J. Appl. Phys. 123, 180901 (2018). Work supported in part by CMRR and DARPA.