TFuzzyScore: A Hybrid Similarity Metric Combining TF-IDF and Fuzzy Logic for Entity Mapping in Structured Text
Abhishek Tyagi · 2025
In enterprise-scale data ecosystems, free-text fields often exhibit inconsistencies due to typographical errors, abbreviations, and format drift, complicating entity resolution tasks. This paper presents TFuzzyScore, a hybrid similarity metric that combines statistical semantic modelling via TF-IDF vectorization with lexical proximity using fuzzy string logic (Levenshtein distance). The method computes a composite similarity score as the average of cosine similarity and normalized fuzzy score, offering both interpretability and robustness. Experiments conducted on a procurement dataset comprising over one million cost object descriptors demonstrated that TFuzzyScore achieved a 67% correct match rate, significantly outperforming SQL rules (6%) and standalone fuzzy logic (13%). The algorithm is implemented in Apache Spark, ensuring scalability for large-scale applications. Theoretical foundations, architecture design, benchmarking, and limitations are discussed, positioning TFuzzyScore as a viable solution for real-world text matching, data harmonization, and master data management in low-context environments.