Non-Hermitian three-mode couplers: Higher-order exceptional points in a lossy 3-leg beam splitter

Classical and quantum photonic systems with exceptional points, where eigenvalues and the corresponding eigenstates coalesce, have attracted tremendous interest due to their intriguing properties, topological features, and an enhanced sensitivity to external perturbations. Non-Hermitian mode-coupling matrices provide a tractable analytic framework to model gain, loss, and chirality across optical, electronic, and mechanical platforms without the complexity of full open-system dynamics. Exceptional points define their spectral topology and enable applications in mode control, amplification, and sensing. We introduce a general algebraic framework for arbitrary N-mode couplers in classical and quantum regimes and develop it explicitly for N=3. This case admits algebraic diagonalization, where a propagation-dependent gauge aligns local and dynamical spectra and reveals the geometric phase between adiabatic and exact propagation. In particular, we study parity-time (PT) symmetric and non-Hermitian cyclic couplers, where two exceptional points of order three lie within a continuum of exceptional points of order two, ruling out pure encircling. As an application, we study these exceptional points for a lossy 3-leg beam splitter and reveal its rich propagation dynamics as a function of initial states (e.g., Fock and NOON states). Our approach provides a systematic route to analyze non-Hermitian mode couplers and guide design in classical and quantum platforms.