Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Tasks
Anwoy Chatterjee, Eshaan Tanwar, Subhabrata Dutta, Tanmoy Chakraborty · 2024
Large Language Models (LLMs) have transformed NLP with their remarkable In-context Learning (ICL) capabilities.Automated assistants based on LLMs are gaining popularity; however, adapting them to novel tasks is still challenging.While colossal models excel in zero-shot performance, their computational demands limit widespread use, and smaller language models struggle without context.This paper investigates whether LLMs can generalize from labeled examples of predefined tasks to novel tasks.Drawing inspiration from biological neurons and the mechanistic interpretation of the Transformer architecture, we explore the potential for information sharing across tasks.We design a cross-task prompting setup with three LLMs and show that LLMs achieve significant performance improvements despite no examples from the target task in the context.Cross-task prompting leads to a remarkable performance boost of 107% for LLaMA-2 7B, 18.6% for LLaMA-2 13B, and 3.2% for GPT 3.5 on average over zeroshot prompting, and performs comparable to standard in-context learning.The effectiveness of generating pseudo-labels for in-task examples is demonstrated, and our analyses reveal a strong correlation between the effect of crosstask examples and model activation similarities in source and target input tokens.This paper offers a first-of-its-kind exploration of LLMs' ability to solve novel tasks based on contextual signals from different task examples.Definition: You are given a passage as context and a question related to the passage that can be answered as "True" or "False".Based on the context, question and your reasoning ability answer in a "True" and "False".Context: Many organisms, the action potential is actually initially carried.... Question: Do all neurons have the same action potential?Answer: False Context: Intersex is in some caused by unusual sex hormones.... Question: Can u be born with both male and female parts?Answer: True Context: The Gregorian leap cycle, which has 97 leap days spread.... Question: Are there ever 53 weeks in a year?Answer: True Definition: Given a question from a scientific exam about Physics, Chemistry, and Biology, among others.The question is in multiple choice format with four answer options A., B., C. and D. Using your knowledge about the scientific fields answer the question and provide the label A, B, C and D as answer.Question: Which of the following is not true for myelinated nerve fibers:A. Impulse through ........ neural fibers B. Membrane ...... of Ranvier C. Saltatory .... is seen D. Local anesthesia ....... when the nerve is not covered by myelin sheath Semantically similar example selectionDefinition: You are given a passage as context and a question related to the passage that can be answered as "True" or "False"......