Time-reversal-symmetry-broken nematic insulators near quantum spin Hall phase transitions

Quantum spin Hall insulators have drawn attention in recent years because they support time-reversal symmetry protected helical edge states. We study the phase diagram of a model quantum spin Hall system as a function of band inversion and band-coupling strength, demonstrating that when the latter is weak, an interaction induced nematic insulator state emerges over a wide range of band inversion. This property is a consequence of the long-range Coulomb interaction, which favors interband phase coherence that is weakly dependent on momentum and is therefore frustrated by the single-particle Hamiltonian at the band inversion point. For weak band hybridization, interactions convert the continuous gap-closing topological phase transition at inversion into a pair of continuous phase transitions bounding a state with broken time-reversal and rotational symmetry, and no gap closing. At intermediate band hybridization the topological phase transition proceeds instead via a Chern insulator state, whereas at strong hybridization interactions play no essential role. We comment on the implications of our findings for InAs/GaSb quantum spin Hall systems.