Data Stream Clustering Using Embedding Dimension at Edge Gateway
Shachi Sharma, Ravi Kothari, Obaidullah Ghaswi · 2022
Data generated by Internet of Things (IoT) devices is typically transmitted to cloud data centers for processing. The huge volume of data generated by many IoT devices prompts exploring the possibility of processing the data at the edge gateway and transmitting summarized information to the cloud data center for more complex processing. Given the limited CPU and memory available at edge gateways, a promising and emerging paradigm is the use of stream based clustering and transmission of the summarized information in the form of cluster centers to the core. Motivated by these considerations, we present in this paper the results of our efforts in constructing a system that processes IoT data based on this promising paradigm. Constructing the system requires choosing a run-time for stream based processing. We find that Apache Storm allows for low latency processing with low memory requirements. We then propose a new algorithm, DenStreamED, that uses the embedding dimension to represent the incoming data into an embedded space in which stream based clustering is performed. Experiments on real data sets, carried out by executing DenStreamED in Apache Storm, show the efficacy of our proposed algorithm and the feasibility of the proposed paradigm in providing superior results while minimizing network traffic and improving scalability.