Advanced Machine Learning Framework for Photon Discrimination in Hybrid Neutrino Detectors

Hybrid detectors based on water-based liquid scintillator (WbLS) technology offer a promising path toward next-generation neutrino experiments by enabling both high light yield and directional sensitivity. A long-standing challenge for such detectors is the reliable separation of Cherenkov and scintillation photons, a task traditionally performed using timing cuts or spectral filters, which are sensitive to detector geometry and often result in a significant loss of photon statistics. In this work, we present a comprehensive machine-learning (ML) framework developed to perform photon-type discrimination under two distinct optical environments: (1) directional beam interaction, representative of accelerator-based neutrino experiments, and (2) fully isotropic interaction, relevant to solar, atmospheric, and supernova neutrinos. Using Geant4 simulations, more than one million photon events were generated, incorporating arrival time, photon energy, and 3-dimensional PMT positions as classifier inputs. More than 20 ML models were benchmarked, followed by systematic hyperparameter optimization of the top performers (XGBoost, LightGBM, Random Forest, CatBoost, and Gradient Boosting). To explore model complementarity, different ensemble combinations were constructed. The optimal ensemble classifier achieved 95.7% accuracy in the directional case and 94% in the isotropic case, improving separation efficiency by ~6% compared to traditional time-cut methods. These results demonstrate that ML-based photon separation is both geometry-independent and highly scalable, offering a robust path toward enhanced particle identification, background suppression, and reconstruction fidelity in future WbLS-based detectors such as ANNIE, EOS, THEIA, and kiloton-scale neutrino observatories.