Relegation classifier: a machine-learning approach for optimizing analysis significance in signal identification

Use of machine learning (ML) models to classify signal/background is a critical component of many analyses in particle physics and astrophysics. In these applications, maximizing the statistical significance ($\sigma$) of the resulting signal sample is often paramount to maximizing classification accuracy, the typical figure of merit for model optimization. However, accuracy and statistical significance can in fact be in tension in applications where signal and background are inseparable in some regions of the input feature space. We present a novel approach to multi-class problems that optimizes a neural network to predict into an expanded category space using a loss function that combines accuracy and statistical significance. This approach allows the model to ignore regions of the input space in which events from multiple classes are impossible to separate without overfitting. We demonstrate the application of the relegation approach to hadronic-physics datasets and show that it produces analysis significance comparable to logistic regression.