Generalizing Data Processing for Natural Language Processing Tasks
Ziqian Luo · HAL (Le Centre pour la Communication Scientifique Directe) · 2024
Natural Language Processing (NLP) contains a wide range of tasks such as sequence tagging, dependency parsing, document classification. With the maturity of deep learning, nowadays a single model can be leveraged for multiple different tasks with more standardized data pre-processing and post- processing. In our project, we designed and implemented a general framework for doing data processing. We also apply our framework for serveral different NLP tasks. Our experiment shows that with the usage of our framework, the data processing code becomes cleaner and simpler with significantly reduction of the number of lines. More importantly, our framework provides a solution to entirely separate data processing from model training and prediction. Such separation will be helpful for doing more complicate tasks such as distribute training.