Theory of moiré magnetism in twisted bilayer α-RuCl₃

We present a comprehensive theory of the long-range ordered magnetic phases of twisted bilayer α-RuCl3 via a combination of first principles calculations and atomistic simulations. While a monolayer exhibits zigzag antiferromagnetic order with three possible ordering wave vectors, moir ́e superlattices show a single domain (1q-1q) zigzag order with different wave vectors on each layer for large twist angles. For small twist angles, the ground state involves a combination of all three wave vectors (3q-3q) in a complex hexagonal domain structure. The multi-domain order minimizes the interlayer energy while enduring the energy cost due to the domain wall formation. Comparing these two effects, we obtain an analytical estimate for the critical angle for the transition from 1q to 3q that is in agreement with atomistic simulations. At intermediate angle, we obtain a 2q domain structure on each layer that accommodates both effects. Our results indicate that magnetic frustration due to stacking-dependent interlayer exchange in moir ́e superlattices can be used to tune the magnetic ground state and enhance quantum fluctuations in α-RuCl3.