A Generic Approach of Filling Missing Values in NCDC Weather Stations Data
Doreswamy Hosahalli, Ibrahim Gad · 2018
Missing data is a common problem in several real applications. Moreover, mainstream solutions to solve the missing data issue either fill in the missing values (imputation) that aim to complete dataset or ignore the missing data (marginalization) but these solutions may have notable costs in the final decision. Imputed values are considered as the same as the actually observed data where the validation of it based on the method used to predict it. In general, machine learning algorithms cannot analyze weather dataset that has missing values. Grouping stations are used to identify the group of similar stations based on the number of missing data in weather datasets, such as NCDC. In contrast, this work presents a new framework for filling missing values in the observed features based on the group of the station, which identifies by the total number of missing data of each station. The proposed method presents a simple way for grouping stations based on the missing data, and the type of group outputted by grouping process is identifying the technique that is used to fill missing values of each station. In experiments on NCDC data, we show that the new system is an effective way to enable imputation of missing values.