Large-scale Lifelong Learning of In-context Instructions and How to Tackle It
Jisoo Mok, Jaeyoung Do, Sung‐Jin Lee, Tara Taghavi, Seunghak Yu, Sungroh Yoon · 2023
Jointly fine-tuning a Pre-trained Language Model (PLM) on a pre-defined set of tasks with in-context instructions has been proven to improve its generalization performance, allowing us to build a universal language model that can be deployed across task boundaries.In this work, we explore for the first time whether this attractive property of in-context instruction learning can be extended to a scenario in which tasks are fed to the target PLM in a sequential manner.The primary objective of so-called lifelong in-context instruction learning is to improve the target PLM's instance-and task-level generalization performance as it observes more tasks.DYNAINST, the proposed method to lifelong in-context instruction learning, achieves noticeable improvements in both types of generalization, nearly reaching the upper bound performance obtained through joint training.