Design and implementation of data integrity check of subway log collection system
Qi Tao, Yuan Li, Youxun Lei · 2016
Analyzing subway operation logs can help the management of the subway. In order to make the analysis results accurate, it is important to ensure the integrity of the collected subway logs. This paper proposes a subway log collection system based on Flume in which the integrity of the collected logs is checked. This paper describes the integrity check module in which HBase is used to store the large volume of logs. Through the preliminary processing of collected data and the design of HBase tables, it implemented the integrity check of collected data. After the integrity check, missing data is found and retransmitted for the following analysis.