Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations
W. Chen, Cheng-Kuang Wu, Yun-Nung Chen, Hsin‐Hsi Chen · 2023
Large language models (LLMs) have exhibited striking in-context learning (ICL) ability to adapt to target tasks with a few inputoutput demonstrations.For better ICL, different methods are proposed to select representative demonstrations from existing training corpora.However, such settings are not aligned with real-world practices, as end-users usually query LMs without access to demonstration pools.In this work, we introduce SELF-ICL-a simple framework which bootstraps LMs' intrinsic capabilities to perform zero-shot ICL.Given a test input, SELF-ICL first prompts the model to generate pseudoinputs.Next, the model predicts pseudo-labels for the pseudo-inputs via zero-shot prompting.Finally, we perform ICL for the test input with the pseudo-input-label pairs as demonstrations.Evaluation on 23 BIG-Bench Hard tasks shows SELF-ICL outperforms zero-shot baselines on both average accuracy and head-to-head comparison.Moreover, with zero-shot chain-ofthought, SELF-ICL achieves results comparable to using real demonstrations.Additionally, we conduct a range of analyses to validate SELF-ICL's effectiveness and provide insights for its behaviors under different settings.1 * Equal contribution. 1 https://github.com/ntunlplab/Self-ICLFollowing is an example instance for the task: Evaluate the result of a random Boolean expression.Please come up with 3 new, diverse, and creative instances for the task.Example instance: Q: not ( True ) and ( True ) is New instance 1: Q: ( False ) or ( False ) and ( True ) is New instance 2: Q: ( True ) and ( False ) or ( False ) is New instance 3: Q: not ( False ) and ( True ) or ( False ) Task description: Evaluate the result of a random Boolean expression.Q: ( False ) or ( False ) and ( True ) is A: False Q: ( True ) and ( False ) or ( False ) is A: False Q: not ( False ) and ( True ) or ( False ) A: True Q: not ( True ) and ( True ) is