Petaminer: Efficient Navigation to Petascale Data Using Event-Level Metadata
A. Vaniachine, Jack Cranshaw, Paul J. Hamill, David M. Malon · 2009
MySQL is a standards-compliant, high-performance, relational database with a pluggable storage engine architecture that enables third parties to create low-level interfaces to data stored in different formats, permitting more optimal operation in various use scenarios.The Petaminer is a joint project with a private company whose purpose is to exploit a feature of MySQL to provide a direct "MySQL interface" to ROOT files.We report on the development of a custom MySQL storage engine to directly access the LHC experiment's event TAG metadata stored in ROOT files.ROOT is a structured, extensible analysis framework and data architecture for storage and retrieval of arbitrarily scaled, distributed data.ROOT supports high-rate streaming of data as well as storage distributed across multiple file systems and domains.It was designed to contain physics event data among other uses, and so has numerous features that make it ideal for this purpose, including hierarchical, column-oriented data structures, dynamic schema evolution, and support for binary float numbers.The ability to access column-oriented event data stored in distributed ROOT files using standard MySQL tools offers the potential to improve the efficiency and accessibility of metadata-keyed queries for LHC data analysis.The Petaminer project demonstrated feasibility of using ROOT TTree files as a pluggable storage engine for MySQL.We present first results of our feasibility studies of creating a columnoriented MySQL storage engine that uses the ROOT API to access TAG metadata directly from ROOT files, and uses ROOT-FastBit to map MySQL indexing operations to FastBit indices.The prototype storage engine demonstrates both query functionality and improved performance.As a result, Petaminer enhances MySQL with column-oriented storage capable to support binary float numbers with bitmap indexing.Work is in progress to extend the functionalities of the Petaminer software.