Towards Making the Most of ChatGPT for Machine Translation
Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, Dacheng Tao · 2023
ChatGPT shows remarkable capabilities for machine translation (MT).Several prior studies have shown that it achieves comparable results to commercial systems for high-resource languages, but lags behind in complex tasks, e.g., low-resource and distant-language-pairs translation.However, they usually adopt simple prompts which can not fully elicit the capability of ChatGPT.In this paper, we aim to further mine ChatGPT's translation ability by revisiting several aspects: temperature, task information, and domain information, and correspondingly propose an optimal temperature setting and two (simple but effective) prompts: Task-Specific Prompts (TSP) and Domain-Specific Prompts (DSP).We show that: ❶ The performance of ChatGPT depends largely on temperature, and a lower temperature usually can achieve better performance; ❷ Emphasizing the task information can further improve ChatGPT's performance, particularly in complex MT tasks; ❸ Introducing domain information can elicit ChatGPT's generalization ability and improve its performance in the specific domain; ❹ ChatGPT tends to generate hallucinations for non-English-centric MT tasks, which can be partially addressed by our proposed prompts but still need to be highlighted for the MT/NLP community.We also explore the effects of advanced in-context learning strategies and find a (negative but interesting) observation: the powerful chain-ofthought prompt leads to word-by-word translation behavior, thus bringing significant translation degradation.