Suri: Multi-constraint Instruction Following in Long-form Text Generation
Chau Minh Pham, Simeng Sun, Mohit Iyyer · 2024
Existing research on instruction following largely focuses on tasks with simple instructions and short responses.In this work, we explore multi-constraint instruction following for generating long-form text.We create Suri, a dataset with 20K human-written long-form texts paired with LLM-generated backtranslated instructions that contain multiple complex constraints.Because of prohibitive challenges associated with collecting human preference judgments on long-form texts, preferencetuning algorithms such as DPO are infeasible in our setting; thus, we propose Instructional ORPO (I-ORPO), an alignment method based on the ORPO algorithm.Instead of receiving negative feedback from dispreferred responses, I-ORPO obtains negative feedback from synthetically corrupted instructions generated by an LLM.Using Suri, we perform supervised and I-ORPO fine-tuning on Mistral-7b-Instruct-v0.2.The resulting models, Suri-SFT and Suri-I-ORPO, generate significantly longer texts (∼5K tokens) than base models without significant quality deterioration.Our human evaluation shows that while both SFT and I-ORPO models satisfy most constraints, Suri-I-ORPO generations are generally preferred for their coherent and informative incorporation of the constraints.1 Corrupt Constraints Main Goal: Write a whimsical and imaginative narrative that centers around a young girl's journey through a fantastical world. Constraints:-Start the narrative with the protagonist feeling bored with her current surroundings.-Employ a tone that blends matter-offact narration with elements of dry humor and irony. Back-Translate Backtranslated Instruction x w (LLM-generated)Main Goal: Write a whimsical and imaginative narrative that centers around a young girl's journey through a fantastical world. Constraints:-Start the narrative with the protagonist without any prior feelings of boredom with her surroundings.-Employ a tone that is consistently serious and devoid of humor.