Finding Persistent Items using Invertible Bloom Lookup Table
Zhoudan Lv, Feng He, Lin Chen · 2019
With the rapid development of information technology, the age of big data requires new data processing methods with higher time and space efficiency. The method of finding persistent items in big data stream becomes an important research direction with promising application field, such as network security or sensor data mining. In this paper, the data processing algorithm is designed to quickly record and accurately identify the persistent items from the big data stream with relatively small memory occupation cost. This algorithm combines the data structures of Invertible Bloom Lookup Table (IBLT) and Bloom Filter (BF) to guarantee the reliability. By simulation and comparison, we prove the low false rate of the algorithm.