Helical versus collinear antiferromagnetic order tuned by magnetic anisotropy in polar and chiral (Ni,Mn)3TeO6

Polar and chiral Ni3TeO6 was found to exhibit a colossal magnetoelectric (ME) effect associated with collinear antiferromagnetic order. We have investigated the evolution of its magnetic state with the substitution of the Ni2+ spins by isotropic Mn2+ spins, namely (Ni,Mn)3TeO6 (NMTO). The ground state of NMTO maintains an Ising nature, but a new phase with XY-type magnetic anisotropy is discovered at an intermediate temperature range. Neutron powder diffraction experiments reveal that (1) Mn ions tend to occupy a specific transition-metal site, consistent with the first-principle calculation, and (2) the intermediate phase is an incommensurate (IC) helical magnetic state propagating along the c axis with q - IC=(0,0,1.5±δ). We also found that the balance between these two magnetic states can be readily manipulated with magnetic fields, which results in significant ME effects. This selective site occupancy of NMTO allows for a unique competition between collinear and helical magnetic structures with minimal chemical disorder. Thus, NMTO serves as a model system to study site-specific chemical control of noncollinear magnetism and ME coupling.