Magnetic phase diagram slightly below the saturation field in the stacked J1-J2 model in the square lattice with the JC interlayer coupling

We study the effect of adding interlayer coupling to the square lattice, $J_1$-$J_2$ Heisenberg model in high external magnetic field. In particular, we consider a cubic lattice formed from stacked $J_1$-$J_2$ layers, with interlayer exchange coupling $J_C$. For the 2-dimensional model ($J_C=0$) it has been shown that a spin-nematic phase appears close to the saturation magnetic field for the parameter range $-0.4\leq J_2/J_1$ and $J_2>0$. We determine the phase diagram for 3-dimensional model at high magnetic field by representing spin flips out of the saturated state as bosons, considering the dilute boson limit and using the Bethe-Salpeter equation to determine the first instability of the saturated paramagnet. Close to the highly frustrated point $J_2/J_1\sim 0.5$, we find that the spin-nematic state is stable even for $|J_C/J_1|\sim 1$. For larger values of $J_2/J_1$, interlayer coupling favors a broad, phase-separated region. Further increase of $|J_C|$ stabilizes a collinear antiferromagnet, which is selected via the order-by-disorder mechanism.