Orthorhombic-to-monoclinic transition in Ta2NiSe5 due to a zone-center optical phonon instability

I study dynamical instabilities in Ta2NiSe5 using density functional theory based calculations. The calculated phonon dispersions show two unstable optical branches. All the acoustic branches are stable, which shows that an elastic instability is not the primary cause of the experimentally observed orthorhombic-to-monoclinic structural transition in this material. The largest instability of the optical branches occurs at the zone center, consistent with the experimental observation that the size of the unit cell does not multiply across the phase transition. The unstable modes have the irreps B1g and B2g. Full structural relaxations minimizing both the forces and stresses find that the monoclinic C2/c structure corresponding to the B2g instability has the lowest energy. Electronic structure calculations show that this low-symmetry structure has a sizable band gap. This suggest that a B2g zone-center optical phonon instability is the primary cause of the phase transition. An observation of a softening of a B2g zone-center phonon mode as the transition is approached from above would confirm the mechanism proposed here. If none of the B2g modes present in the material soften, this would imply that the transition is caused by electronic or elastic instability.