Zero-Shot Prompting for LLM-Based Machine Translation Using In-Domain Target Sentences
Baijun Ji, Xiangyu Duan, Yue Zhang, Kaixin Wu, Min Zhang · IEEE Transactions on Audio Speech and Language Processing · 2024
One promising aspect of using large language models (LLMs) for translation is their ability to adapt effectively to unseen domains without fine-tuning, through strategic prompting. However, the success of this approach largely depends on the selection of relevant examples from domain-specific databases to construct the few-shot prompt for the given input. This work propose a novel zero-shot technique that circumvents data limitations by generating synthetic translation examples from monolingual domain-specific target texts. Our preliminary experiments suggest that LLMs demonstrate increased sensitivity to noise on the target side of examples, while maintaining a tolerance for variations in example quality. Based on these findings, we generate pseudo-examples by using a cross-lingual sentence encoder to identify sentences in the target language that correspond to the input, and then pairing them with coarse translations derived from a dictionary. To improve the retrieval of better pseudo-examples for translation tasks, we enhance the sentence encoder by incorporating domain information through conditional layer normalization and introducing two innovative training objectives: LLM Preference Prediction and Windowed Sentence Prediction. Experiments conducted across various domains demonstrate that our approach not only significantly outperforms previous zero-shot methods but is also comparable to the robust NLLB-3.3B and ALMA-13B(-R) model.