Bringing Data Analysis to the Files and the Database to the Command Line
Saikiran Potti, Andrew Nguyen, Lance Gibson, Antonio Badia · 2023
The analysis of a dataset starts with exploratory and lightweight analysis, in order to gain a rough understanding of the data under consideration. While several tools already exist to support this task, most require that data be loaded and encoded before starting. We propose a set of tools to carry out an initial analysis directly on files of raw data. In particular, we propose a tool that supports simple SQL queries directly over files without the need for any database system. Our implementation is based on expanding some basic capabilities of the Unix command line and exploiting its ability to compose sequences of commands in pipelines. We present an initial implementation and report on some preliminary results.