SparkFlow: A Simple Big Data Analysis Application over Apache Spark
Henry Novianus Palit, Andre Fuad Gunawan, Daniel Jeremia · 2024
With the rapid increase in data generated through our daily activities, it is only a matter of time before we need to analyze this data to support decision making and improve our lives. While many frameworks and tools are available for processing Big Data, most require technical knowledge and skills that can be challenging for common people to acquire. To bridge this gap, SparkFlow was developed with a focus on simplicity and user-friendliness. Built on top of Spark, a well- known Big Data framework capable of processing data promptly and efficiently, SparkFlow features visual block programming that allows non-technical users to easily create data analysis workflows. SparkFlow's elements then convert these workflows into Spark operations for execution on the underlying cluster. Based on the evaluation results, SparkFlow can effectively reduce the number of required steps by at least 30% and accelerate task completion by approximately 13%. Additionally, most users (4 out of 5) find it helpful, easy to use, and beneficial.