Introducing a New Client/Server Framework for Big Data Analytics with the R Language
Drew Schmidt, Wei-Chen Chen, George Ostrouchov · 2016
Historically, large scale computing and interactivity have been at odds. This is a particularly sore spot for data analytics applications, which are typically interactive in nature. To help address this problem, we introduce a new client/server framework for the R language. This framework allows the R programmer to remotely control anywhere from one to thousands of batch servers running as cooperating instances of R. And all of this is done from the user's local R session. Additionally, no specialized software environment is needed; the framework is a series of R packages, available from CRAN. The communication between client and server(s) is handled by the well-known ZeroMQ library. To handle server side computations, we use our established pbdR packages for large scale distributed computing. These packages utilize HPC standards like MPI and ScaLAPACK to handle complex, tightly-coupled computations on large datasets. In this paper, we outline the new client/server architecture components, discuss the pros and cons to this approach, and provide several example workflows that bring interactivity to potentially terabyte size computations.