Towards Reliable Memory Management for Python Native Extensions
Joannah Nanjekye, David Bremner, Aleksandar D. Micić · 2023
Many programming languages provide a C interface as a foreign function interface (FFI) for C developers to access the language, the Python language being one of these languages. Over the years, the Python C API has grown to be a challenge for the evolution of the Python ecosystem. In this paper, we implement a new Python FFI, we call CyStck, by combining a stack and light-weight handles, to support efficient garbage collection (GC) in Python native extensions. CyStck introduces execution time overhead while copying the fewest bytes for all benchmarks across the CPython boundary compared to the CPython API and HPy respectively. We also implemented a tool to automate the migration of extensions from the CPython C API to CyStck using advanced pattern matching and static analysis, with a success rate as high as 90% in some workloads.