Instruction-Tuning LLMs for Event Extraction with Annotation Guidelines
Saurabh Kr. Srivastava, Sweta Pati, Ziyu Yao · 2025
In this work, we study the effect of annotation guidelines-textual descriptions of event types and arguments, when instruction-tuning large language models for event extraction.We conducted a series of experiments with both human-provided and machine-generated guidelines in both full-and low-data settings.Our results demonstrate the promise of annotation guidelines when there is a decent amount of training data and highlight its effectiveness in improving cross-schema generalization and low-frequency event-type performance.1 * The first two authors contribute equally. 1 Our source code and datasets are available at https://github.com/Ziyu-Yao-NLP-Lab/PyCode-TextEE.# This is an event extraction task ... # The following lines describe the task definition @dataclass class Extradite(JusticeEvent): mention agent person destination # This is the text to analyze After getting caught they were transferred to the U.S. for trial.text = result = [ ] Event Schema Code PromptYou are an expert in annotating NLP datasets for event extraction.Your task is to generate annotation guidelines for the event type Extradite which is a child event type of super class JusticeEvent.The event schema is as follows:The following examples are negative examples, as they illustrate different event types provided for contrast and differentiation:The below examples are positive examples, as they match the Event Type being annotated ### Instructions ### 1. Identify and List All Unique Arguments.2. Define the Event Type: Write 5 clear and specific definitions, starting with "The event is triggered by ...": 3. Define Each Argument:** For each argument, provide 5 definitions. Guideline Generation Prompt (Guideline-PN)10 Positive Event Samples 15 Negative Event Samples Event Trans -port Movement Extra dite Convict Justice ... Event Ontology