Scrybe: Enabling Programmatic Interfaces for Explorations Over Voluminous Spatiotemporal Data Collections
Kassidy Barram, Sangmi Lee Pallickara, Shrideep Pallickara · 2024
This study focuses on enabling programmatic interfaces to perform exploratory analyses over voluminous data collections. The data we consider can be encoded in diverse formats and managed using diverse data storage frameworks. Our framework, code named Scrybe, manages the competing pulls of expressive computations and the need to manage resource utilization in shared clusters. The framework includes support for differentiated quality of service allowing preferentially higher resource utilization for certain users. We have validated our methodology with voluminous data collections housed in relational, NoSQL/document, and hybrid storage systems. Our performance benchmarks profile several aspects of our methodology, and demonstrate the effectiveness of our methodology.