Simultaneous observation of super-Heisenberg scaling and spin squeezing in a nonlinear measurement of atomic spins

We report a nonlinear alignment-to-orientation conversion (AOC) [PRL 85, 2088 (2000)] measurement of atomic spins that simultaneously shows super-Heisenberg scaling and achieves projection-noise limited sensitivity. Using this technique, we have recently demonstrated conditional spin squeezing of the atomic ensemble, and entanglement-enhanced measurement sensitivity useful for optical magnetometry [PRL 109, 253605 (2012)]. In addition, we use a novel technique to explicitly certify that the measurement fulfills all the conditions required for quantum non-demolition measurement [NJP 14, 085021 (2012)], which is non-trivial in large spin (J $>$ 1/2) systems. Lastly, we demonstrate that the measurement shows super-Heisenberg scaling with photon number due to the nonlinearity of the AOC technique. This scaling was recently demonstrated in a proof-of-principle experiment [Nature 471, 486-489 (2011)], however in this experiment the measurement sensitivity was more than an order of magnitude worse than the projection noise limit. Here we achieve a sensitivity (observed read-out noise) of 990 spins, competitive with the best observed sensitivity in an equivalent linear measurement [PRL 104, 093602 (2010)], and 20 dB more sensitive than the previous best nonlinear measurement.