Anisotropic quantum spin Hall effect, spin-orbital textures, and the Mott transition

We investigate the interplay between topological effects and Mott physics in two dimensions on a graphenelike lattice, via a tight-binding model containing an anisotropic spin-orbit coupling on the next-nearest-neighbor links and the Hubbard interaction. We thoroughly analyze the resulting phases, namely, a topological band insulator phase or anisotropic quantum spin Hall phase until moderate-strength interactions and a Néel and a spiral phase at large interactions in the Mott regime, as well as the formation of a spin-orbital texture in the bulk at the Mott transition. The emergent magnetic orders at large interactions are analyzed through a spin-wave analysis and mathematical arguments. At weak interactions, by analogy with the Kane-Mele model, the system is described through a Z₂ topological invariant. In addition, we describe how the anisotropic spin-orbit coupling already produces an exotic spin texture at the edges. The physics at the Mott transition is described in terms of a U(1) slave-rotor theory. Taking into account gauge fluctuations around the mean-field saddle-point solution, we show how the spin texture now proliferates into the bulk above the Mott critical point. The latter emerges from the response of the spinons under the insertion of monopoles and this becomes more pronounced as the spin-orbit coupling becomes prevalent. We discuss implications of our predictions for thin films of the iridate compound Na₂IrO₃ and also graphenelike systems.