Competing weak localization and weak antilocalization in ultrathin topological insulators

We demonstrate the evidences of a surface gap opening in (Bi$_{\mathrm{0.57}}$Sb$_{\mathrm{0.43}})_{\mathrm{2}}$Te$_{\mathrm{3}}$ samples for film thickness below 6 quintuple layers, through magnetotransport and scanning tunneling spectroscopy measurements. By tuning Fermi level position relative to the gap, the striking crossover between weak antilocalization and weak localization is observed in nonmagnetic 4 and 5 QL films at low field region, a characteristic feature of quantum interferences competition, possibly owing to the change of net Berry phase. Furthermore, when the Fermi level is swept into the surface gap, the overall unitary behaviors are revealed at higher magnetic field, which are in contrast to the pure WAL signals obtained in thicker films. Besides, the surface bandgap of ultrathin film is also determined by low temperature STS measurements. Our findings show an exotic phenomenon characterizing the gapped TI surface states and point to the future realization of quantum spin Hall effect and dissipationless TI-based applications.