Magnetic phases in the triangular lattice realized in Sn on Si(111) monolayers

Motivated by the experimental realization of a Hubbard model at half-filling with multiple neighbor hoppings in a triangular lattice when Sn is doped in Si (111) monolayers, we study the magnetic phases of an effective spin-Hamiltonian with direct and ring-exchange terms and up to third nearest neighbor interactions derived from first principles calculations. Previous numerical studies on the phase diagram of a spin-Hamiltonian indicate that the material lies in a regime close to the boundaries of collinear (single-Q ordered), chiral spin liquid and tetrahedral (triple-Q ordered) phases. We show that the numerically accurate determination of the magnetic phase of the material is subject to strong finite size effects and boundary conditions of the cluster used in simulations. We revisit the problem using DMRG algorithm on larger clusters and consider different boundary conditions to determine the magnetic phase of the material in the thermodynamic limit. We use the chiral order parameter and the spin-structure factor to identify and characterize the competing magnetic phases in the effective spin model.