Massive AIS Data Management Based on HBase and Spark
Jiwei Qin, Liangli Ma, Jinghua Niu · 2018
With the popularization of AIS technology on ships, the scale of maritime traffic data are rapidly increasing. The traditional management system of AIS data based on relational databases is faced with difficulties in expansion and low access efficiency. In view of that, we present AISHS, a distributed system based HBase and Spark to offer efficient storage and near-real-time query to massive AIS Data. In order to avoid the communication overhead caused by regroup the trajectory in query process, AISHS stores all data of a ship in one Region of HBase, constructs a global secondary index by the spatial-temporal attributes of AIS data. Based on this, we implement the co-location between HBase Regions and Spark RDD partitions to ensure efficient spatial-temporal query. Experiments on real datasets show that AISHS is suitable for the storage and query of AIS data.