Theoretical study of topological phase transitions in (Bi$_{1-x}$In$_{x})_2$Se$_3$ and (Bi$_{1-x}$Sb$_{x})_2$Se$_3$

We use first-principles calculations to study the phase transition from a topological to a normal insulator with concentration $x$ in (Bi$_{1-x}$In$_{x})_2$Se$_3$ and (Bi$_{1-x}$Sb$_{x})_2$Se$_3$ in the Bi$_2$Se$_3$ crystal structure. The spin-orbital coupling (SOC) strength is similar in In and Sb, which have similar atomic numbers, so that if the topological transitions in (Bi$_{1-x}$In$_{x})_2$Se$_3$ and (Bi$_{1-x}$Sb$_{x})_2$Se$_3$ are purely driven by the decrease of SOC strength, we would expect to see similar critical concentrations $x_{\rm c}$ in the two systems. However, based on our preliminary calculations, $x_{\rm c}$ is much lower in (Bi$_{1-x}$In$_{x})_2$Se$_3$ than in (Bi$_{1-x}$Sb$_{x})_2$Se$_3$, indicating that different mechanisms control the behavior in the two cases. Specifically, in (Bi$_{1-x}$Sb$_{x})_2$Se$_3$ we find that the phase transition is mostly dominated by the decrease of SOC. However, for (Bi$_{1-x}$In$_{x})_2$Se$_3$, the In $5s$ orbitals also play an important role, both in the phase-transition behavior and in determining the indirect bulk band gap. Finally, we discuss the accuracy of the energy-level position of the In $5s$ orbitals in (Bi$_{1-x}$In$_{x})_2$Se$_3$ as predicted by density-functional theory and more advanced methods.