Impact of information security measures on the velocity of big data infrastructures
Lionel Dupre, Yuri Demchenko · 2016
Encryption is often viewed as a major drawback which hinders the performance of processing systems. This perception is not wrong; encrypted storage, memory and communications usually perform much slower than systems which process data in the clear. Big Data applications is no exception to the rule: it was designed with Volume and Velocity requirements in mind, and security (i.e. encryption) was initially not considered; perimeter security was deemed sufficient, and Big Data systems were confined to back-end operations. Considering the recent developments in the field (AES-NI processors, Key Management Servers, homomorphic encryption), the encryption vs performance paradigm needs to be actually measured to re-evaluate preconceived reservation. This research found that encryption is no longer an obstacle to efficient and fast Big Data processing, thanks to massive processing parallelisation (which distributes also the encryption payload), new CPU technologies which allow encryption instructions to perform much faster, the use of SSD storage, and finally the clever data-centric use of encryption in HDFS. The paper provides analysis of four strategies in using data encryption in Hadoop based Big Data applications, which have been tested on the testbed built on Amazon Web Services (AWS) platform using advanced AWS monitoring data. Tests were performed on datasets of relatively modest size (about 5-20 Gigabytes), and performance was measured as all data could fit in each node's RAM. On larger datasets (e.g. of Terabytes scale), data partitioning may be required to obtain similar results.