Compressed domain-specific data processing and analysis
Dapeng Dong, John Herbert · 2017
Domain specific data such as sensor outputs and server trace logs have low levels of symbol richness, and so they can be represented in a very compact format. In this paper, we present a bit-oriented compression scheme designed not only to represent the data compactly but also to allow MapReduce programs to perform analysis and processing directly on the compressed data, and to do so in parallel. The core of the compression scheme is a novel hybrid data structure supporting bit pattern searching in constant time, and a scheme for making a block-splittable compressed file. Supporting software allows developers to work transparently with the compressed data. Experimental results demonstrate that the proposed compression scheme can significantly reduce data size and improve MapReduce analysis performance.