Exploratory Data Analysis with Database-backed Dataframes: A Case Study on Airbnb Data
Phanwadee Sinthong, Michael J. Carey · 2021 IEEE International Conference on Big Data (Big Data) · 2021
Choosing between various scalable dataframe libraries can be an overwhelming task for data scientists but it is critical because each framework deploys a different optimization technique that could affect the overall performance. Comparing each framework on a set of analytical tasks in isolation might not fully represent the unique characteristics of big data analyses. This paper describes a case study of applying PolyFrame, a database-backed dataframe library, on an end-to-end exploratory data analysis involving Airbnb data. PolyFrame is a scalable data analytics library that provides a Pandas-like dataframe interface on top of a variety of database systems. The familiarity of its interface enables data scientists to interact with large collections of data through a scale-independent data analysis experience without needing significant database or distributed systems knowledge. Throughout this case study we also highlight the scalability benefits and limitations of database-backed dataframes via a performance comparison with Pandas dataframes for each of the stages of the analysis.