Unified Cloud-Native Metadata Discovery with Trino, Superset and Ibis in the LZ Dark Matter Experiment

LUX-ZEPLIN (LZ) is the world’s most sensitive dark matter direct-detection experiment, acquiring petabytes of data per year using a dual-phase xenon time projection chamber (TPC) with a 7 tonne active mass. Metadata related to TPC conditions and data processing environments is stored in six different SQL and NoSQL databases, which are accessed by five bespoke programmatic and graphical interfaces that connect to unique subsets of these databases. The fragmented nature of these interfaces is difficult to maintain and confusing for end-users. 

This presentation details how we deployed Trino and Superset using Helm to provide a unified graphical and SQL interface to access LZ metadata databases. This allowed us to immediately deprecate one interface we maintain, and improve response times by a factor of 4-100x for common queries. I will also discuss our adoption of Ibis to create composable SQL queries using a dataframe-like python API, which promotes their re-use and adoption by end-users for more complex workflows.