Long-term instabilities and unphysical drifts of AI weather models: Challenges of learning multi-scale dynamics

AI weather models, deep neural networks trained only on observation-derived data (reanalysis), have shown remarkable forecast skills for up to 10 days, even outperforming the state-of-the-art numerical weather prediction models. However, these models have been found to go unstable (blow up) or drift to unphysical circulation patterns when run for weeks and months. These models have also been found to represent the small scales (high wavenumbers) incorrectly. Here, we show that the two problems are related, and due to a well-known inductive bias of neural networks called spectral bias. When applied to nonlinear multi-scale dynamical systems such as turbulence and climate, spectral bias leads to errors in small scales, which then grow and accumulate over time and affect all scales. We propose a new algorithm, FouRKS (Fourier regularized loss function with Runge-Kutta integration and Self-supervising layer), which significantly reduces the spectral bias and enables long-term stable, physically consistent integrations. We demonstrate the performance of FouRKS on two-layer quasi-geostrophic turbulent flows and global weather (reanalysis).