To token or not to token: A Comparative Study of Text Representations for Cross-Lingual Transfer
Md. Mushfiqur Rahman, Fardin Ahsan Sakib, Fahim Faisal, Antonios Anastasopoulos · 2023
Choosing an appropriate tokenization scheme is often a bottleneck in low-resource crosslingual transfer.To understand the downstream implications of text representation choices, we perform a comparative analysis on language models having diverse text representation modalities including 2 segmentationbased models (BERT, mBERT), 1 image-based model (PIXEL), and 1 character-level model (CANINE).First, we propose a scoring Language Quotient (LQ) metric capable of providing a weighted representation of both zero-shot and few-shot evaluation combined.Utilizing this metric, we perform experiments comprising 19 source languages and 133 target languages on three tasks (POS tagging, Dependency parsing, and NER).Our analysis reveals that image-based models excel in cross-lingual transfer when languages are closely related and share visually similar scripts.However, for tasks biased toward word meaning (POS, NER), segmentation-based models prove to be superior.Furthermore, in dependency parsing tasks where word relationships play a crucial role, models with their character-level focus, outperform others.Finally, we propose a recommendation scheme based on our findings to guide model selection according to task and language requirements.