Efficient Embedding of Dynamic Languages in Big-Data Analytics
Luca Salucci, Daniele Bonetta, Walter Binder · 2016
Over the last years several frameworks have emerged in the field of big-data analytics. Recent frameworks expose a developer-friendly API via dynamic languages such as Python. Unfortunately, the integration of dynamic languages with the parallel and distributed runtime of such frameworks is cumbersome, as it requires the integration of two or more language virtual machines via inter-process communication, introducing communication overheads and reducing the benefits of the shared memory present in modern multicore machines. In this paper we highlight the advantages of hosting multiple language runtimes in a single shared (language) virtual machine, and the possible performance gain of such an approach in the context of the Apache Spark framework.