Automatic Feature Weighting and Selection for Similar Duplicate Record Detection Based on Ant Colony Optimization
Jianjun Cao, Yuxin Xu, Zhixian Zeng, Chumei Gu, Mengda Wang · 2021
To solve the problem of attributes similarity weighting and similarity selecting towards records for similar duplicate record detection in structured data sources, a feature weighting mathematical model and a multi-object optimization model for feature weighting and selecting are built. The models are solved by a continuous domain ant colony algorithm based on a sliding window grid partitioning strategy. First separately set the search window for the two variables of attribute rank and detection threshold, and combine the two variable windows to form the search space for ant colony optimization. Then balance the heuristic information of different variables by grid width ratio, and use ant colony algorithm to perform the search for the optimal solution. Finally, the optimal solution is used as the center of the window for window sliding, and the search for the optimal solution and window sliding are performed again until the window sliding does not change the optimal solution. By applying the method to engineering practice, the verified that the continuous domain ant colony algorithm based on the grid partitioning strategy of sliding window in similar duplicate record detection.