Phase diagram of the Kitaev-Heisenberg-J₂-J₃ model: A DMRG study

The Kitaev-Heisenberg model is important in understanding the physics of iridates A₂IrO₃ (A=Na,Li). To understand the spin liquid and vairous novel magnetic ordered phases, we investigate the Kitaev-Heisenberg-J₂-J₃ model on the honeycomb lattice by large-scale density-matrix renormalization group simulation. Without J₂, we find that a tiny J₃∼0.04 interaction is enough to drive the system from the Kitaev spin liquid to the zigzag magnetic ordered phase. When J₃=0, the system undergoes a phase transition to a spiral magnetic order phase for relatively larger J₂∼0.15. The full phase diagram of the Kitaev-Heisenberg model in the presence of both J₂ and J₃ is obtained and the connection to iridates is also discussed.