A Comprehensive Test Plan for Natural Language Processing Preprocessing Functions
P. K. Majumdar · American Journal of Information Science and Technology · 2025
This paper outlines a comprehensive testing strategy for validating key natural language processing (NLP) preprocessing functions, specifically preprocess() and get_tokens(). These functions are vital for ensuring high-quality input data in NLP workflows. Recognising the influence of preprocessing on subsequent model performance, the plan employs a layered testing approach that includes functional, edge-case, negative, and property-based tests. It emphasises goals such as ensuring functional correctness, robustness, semantic integrity, and idempotency, supported by thorough test cases and automation with pytest and hypothesis. By systematically tackling pipeline fragility, this framework aims to ensure the reliability and reproducibility of NLP preprocessing, laying the groundwork for dependable, production-ready language models.