Scalable Storage of Sensor Data for a High Frequency Measurement Vehicular Monitoring System
Erik Eliasson, Daniel Öhrlund · Epubl LTU · 2012
This thesis investigates how to store sensor data for a large scale vehicular monitoringsystem with high frequency measurements. The system is intended to measure multiplesensors at a rate of one measurement per second and allow for non-real-time analysisof the data. The goal is that the system should be able to support all cars in Sweden,which means that 240 TB of raw sensor data must be stored every year. To be ableto keep up with the new that is data generated by cars, sensor batches (a batch is acollection of sensor measurements captured at the same time by a single car) must alsobe inserted at a rate of at least 110 000 per second.Cassandra is identifed to be the database that best suits the system requirements.It is then investigated how to store and index the sensor data in Cassandra to be ableto perform one- and two-dimensional range queries. The data models focus on how tostore the data so that it is equally distributed over the cluster and is able to scale upfor large numbers of records.An empirical study is performed that evaluates the performance for insertion, retrievaland storage for the different data models. The result is then used to determinehow many servers would be required to meet the system requirements. The study showsthat it is feasible to store the data in Cassandra. Less than 21 servers are required toreach the insertion rate and the data can be stored using between 360 and 1260 servers(with 2 TB of storage each) per year with three replicas of each node, depending onthe indexing approach.