Unified Cloud-Native Metadata Discovery with Trino, Superset and Ibis in the LZ Dark Matter Experiment
ORAL
Abstract
LUX-ZEPLIN (LZ) is the world’s most sensitive dark matter direct-detection experiment, acquiring petabytes of data per year using a dual-phase xenon time projection chamber (TPC) with a 7 tonne active mass. Metadata related to TPC conditions and data processing environments is stored in six different SQL and NoSQL databases, which are accessed by five bespoke programmatic and graphical interfaces that connect to unique subsets of these databases. The fragmented nature of these interfaces is difficult to maintain and confusing for end-users.
This presentation details how we deployed Trino and Superset using Helm to provide a unified graphical and SQL interface to access LZ metadata databases. This allowed us to immediately deprecate one interface we maintain, and improve response times by a factor of 4-100x for common queries. I will also discuss our adoption of Ibis to create composable SQL queries using a dataframe-like python API, which promotes their re-use and adoption by end-users for more complex workflows.
*This work is supported by the US DOE Office of Science, Office of High Energy Physics; the U.K. Science & Technology Facilities Council; Portuguese Foundation for Science and Technology; the Institute for Basic Science, Korea; the Swiss National Science Foundation; and the Australian Research Council Centre of Excellence for Dark Matter Particle Physics.
–
Presenters
-
Eli Mizrachi
- SLAC National Accelerator Laboratory