Machine learning large-scale simulation and band-mixing fractional quantum anomalous Hall effect in twisted MoTe2

The emergence of topological flat bands in long-wavelength moiré superlattices provides exciting opportunities to realize the lattice analogs of both the integer and fractional quantum Hall effects without external magnetic fields. Recently, optical and direct transport evidence of both integer and fractional quantum anomalous Hall effects have been reported in twisted MoTe2. Here we investigate the moiré band structures and the strong correlation effects in twisted MoTe2 for a wide range of twist angles, employing a combination of various techniques. We first develop the neural network mapping of lattice potential and density functional Hamiltonian under various van der Waals corrections, using massive local training datasets and transfer learning. Using machine-learning accelerated large-scale first principles calculations, we obtain angle dependent topological bands and realistic continuum modeling descriptions down to small twist angles. Furthermore, we explore the phase diagrams and transition of the system through continuum model exact diagonalization. Our multi-band exact diagonalization analysis reveals significant band-mixing effects and the strong competition between charge density wave orders and fractional quantum anomalous Hall states.