A Deletion Aware Usable Space Control for SD2
Joichiro Kon, Kenji Nakashima, Saneyasu Yamaguchi · 2017
Big data processing is one of the most important topics in today's information technology environment. Constructing a secure content distribution platform is necessary for utilizing private data, such as personal genome data. For this purpose, secure data sharing and distribution (SD2) platform for integrated big data utilization was proposed. Private data can be analyzed with encryption on the platform. However, encryption of data sometimes greatly increases the amount of data. Thus, improving I/O performance is important for processing big data securely in a practical amount of time. Hadoop is one of the most popular methods for high-speed big data processing. In this study, we focus on I/O performance of Hadoop. First, we introduce and discuss our existing method for improving Hadoop I/O performance by controlling the file placing location. The method controls the behavior of filesystems and actively utilizes the fast area of storage devices. However, the method does not consider the deletion of files and therefore cannot make use of the faster zones after file removal. Second, we propose a method that considers the areas obtained by deleting files. This method dynamically reduces the usable area to force the fast areas to be used first.