RSECB: A Reed Solomon Erasure Coding System for Hadoop Clusters
P. Joseph Sylvan · International Journal for Research in Applied Science and Engineering Technology · 2018
In this paper we propose a coding technique known as Reed Solomon Erasure Coding using Backblaze method (RSECB) for Hadoop data archival system for Hadoop clusters where RS (m+n; m) codes are employed to archive data replicas in the Hadoop distributed file system (RSECB). We have originated two strategies for archiving RSECB Hadoop system (i.e., Grouping & Pipeline) to speed up the data archival process. RSECB-Grouping: which is based on Map Reduce data archival scheme which stores local key-value in the form of mapped intermediate output Key-Value pairs. With the local store in place, RSEC-Grouping does integration into a single key-value pair with the same key using all intermediate key-value pairs, succeeded by shuffling the single Key-Value pair to reducers to generate final parity blocks. RSECB-Pipeline uses multiple data nodes in a Hadoop cluster for forming data archival pipeline. RSECB-Pipeline conveys the merged single key-value pair to a successive node's local key-value store. In the pipeline, the rearmost node is accountable for outputting parity blocks. We implement RSECB in a real-world Hadoop cluster. The experimental results show that RSECB-Grouping and RSECB-Pipeline diminishes phases by a factor of 15 and 8, subsequently and also accelerates Threshold's shuffle. When block size is larger than 32MB, RSECB Hadoop distributed file system boosts the efficiency of EC and RAID roughly by 16.2% and 42.5%, respectively.