Various Data Skewness Methods in the Hadoop Environment
Subhankar Mishra, Namita Sethi, Ayes Chinmay · 2019
Hadoop provides an environment for efficient storage and processing of data. Time for completion of a BigData job depends on the slowest mapper or slowest reducer. So for an efficient job there has to be an efficient division of data at both mapper side and reducer side which would effectively decrease the time taken by the job to complete. But under many conditions, Hadoop MapReduce fails to implicitly divide the data in an efficient way between mappers and reducers. This gives rise to data skewness or load imbalance. From the study, it is revealed that data skewness can occur at both the mapper side and reducer side, such as the page rank algorithm or cloudburst. Techniques like LEEN, PTSH, SkewTune, Skew Reduce are implemented to tackle the problem of data skewness. Each technique has its own set of advantage and disadvantage. LIBRA and DREAMS are the most preferred and popular techniques against data skewness.