Micromagnetic Insights into History-Dependent Domain Behavior in MBE-grown Magnet/Topological Insulator Heterostructure Fe₃GeTe₂/Bi₂Te₃

Heterostructures of two-dimensional van der Waals magnets and topological insulators are promising platforms for spin‑torque switching, the quantum anomalous Hall effect, and chiral spin textures. Among them, Fe₃GeTe₂/Bi₂Te₃ thin films grown by molecular beam epitaxy stand out for their high spin-orbit torque efficiency and robust interfacial coupling. Using scanning transmission X-ray microscopy, we observe speckled magnetic domains with radii around 50 nm, stable across thicknesses ranging from 1–5 folds of 5-layer Bi₂Te₃ and 8-layer Fe₃GeTe₂, and temperatures of 75 K, 120 K, and 165 K, which is previously unreported from exfoliated single crystal Fe₃GeTe₂. To investigate their origin, we performed micromagnetic simulations with varying Dzyaloshinskii–Moriya interaction (DMI), exchange stiffness, and perpendicular anisotropy. Comparing the experiment with the simulations, we identify the role of DMI in modulating the balance between short and long range magnetic interactions by analyzing domain size and shape with respect to the magnetic field. As DMI dominates more over anisotropy and exchange stiffness, magnetization switching is softened, and the domain evolves from uniform reversal with circular domains to fragmented speckled patterns and, eventually, to stripe-like textures. Finally, we present a phase diagram of domain structures as functions of key parameters, including magnetic field, DMI, anisotropy, and exchange stiffness.