Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform
Yi‐Wei Huang, Sheng-Tzong Cheng · 2018
Recent advances in sensor technology has led to varieties of sensors. Problems facing now is how to exploit and compute these data efficiently. Map Reduce is a programming model to solve these problem. However, it may cause I/O bottleneck. In-memory computing (IMC) comes up to solve the problems. However, network bandwidth remains a bottleneck. It restricts the speed of receiving the information from the source and dispersing information to each node. To our observation, some data from sensor devices might be similar due to temporal or spatial dependency. Therefore, compression technology could be an effective solution. It replaces data with smaller sizes of streams for improving data utilization. This study presents an effective reduce transmission scheme on a distributed real-time IMC platform-Spark Streaming. We design and implement a system to provide a high compression ratio in a small batch data from the source. It is expected to reduce data transmission with little delay time in the soft real-time fashion.