Flexible and efficient computation in hybrid oscillatory and spiking neuronal networks for edge computing applications

Brain circuits flexibly adapt computations “on the fly” and perform energy-efficient computations. Such features may emerge from a synergy between spiking and analog dynamics. The latter includes oscillatory dynamics attributed to computational coordination processes [1]. Here, we propose that artificial networks based on such hybrid dynamics combining brain-inspired coupled oscillator networks with energy-efficient spiking networks can implement attention mechanisms underlying state-of-the-art AI models while offering higher computational flexibility, faster inference, and prolonged history dependence. We first propose a novel hybrid neural network (HNN) architecture and show that the oscillatory component enables flexible information processing in these networks, as well as mitigates the vanishing gradient problem. We demonstrate that for long sequence learning tasks, including the sequential MNIST task, our HNN approach outperforms standard recurrent networks. We further implement our architecture in energy-efficient superconductor hardware designs using a combination of single flux quantum spikes as the computing elements and Josephson-based oscillators to maintain long-term dependencies.

Our approach provides a novel energy-efficient AI tool for fast online processing of sequence data with long-range dependencies. It has an extensive range of applications, particularly where data streams need to be rapidly analyzed, to facilitate decision-making, or to provide actionable insights. Possible applications include robotics or neural prostheses. Our superconducting HNN hardware can enable real-time analysis of data from cryogenic sensors, including microwave or magnetic sensors, as well as control and readouts from cryogenic quantum computers. Our HNN approach also provides an excellent starting point to address the pressing challenge of coordinating large-scale computations, for example, in a mixture of expert systems.

[1] Kirst et al. Nature communications 2016.