Errors Detection Mechanism in Big Data
A. E. M. Eljialy, Sultan Ahmad · 2019 International Conference on Smart Systems and Inventive Technology (ICSSIT) · 2019
The rapid growth of information technologies has not been exempt from errors in the transmission and storage of data, which has affected the services, offered to customers on big data platforms. In these platforms, it has been particularly difficult to identify the causes given their complex nature, the large size and the growth of the data set. The detection and correction of errors in the processing of information in big data platforms are essential to guarantee the integrity of the stored data and thus give users confidence that the information is correctly stored without alterations. However, the options that exist to detect errors of this type lack efficiency in the methods used or use third party software that does not always allow determining the cause of the errors. In this research, big data analysis models are used to find bottlenecks of data processing management. Further, it is developed an alternative error detection mechanism that is based on analyzing error-log file with a variation of Fuzzy clustering mechanism. This approach, based on text mining, helps to look for error frequency and entity that had caused it. Moreover, it is shown how the errors that are presented are distributed according to the source.