Optimizing Data Science Applications using Static Analysis
Bhushan Pal Singh, Mudra Sahu, S. Sudarshan · 2021
Data science applications are often coded in Python, using Pandas and similar APIs. Pandas requires data to be in memory, and when run on larger datasets, these applications may run out of memory, or suffer from poor performance. We describe the SCIRPy system for optimizing such applications by source to source transformations, using static analysis and transformation rules. SCIRPy implements a number of optimizations like data selection, drop column removal, multistage data fetch, and efficient data representation based on metadata analysis. The application source code is transformed into a custom-built intermediate representation (IR) and these optimizations are performed in this IR. The optimized IR is then transformed back to Python source. Our experiments show that our approach reduces the memory footprint and time consumption of a number of data science applications.