Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models
Leonardo Ranaldi, Andre Freitas · 2024
The alignment of reasoning abilities between smaller and larger Language Models are largely conducted via supervised fine-tuning using demonstrations generated from robust Large Language Models (LLMs).Although these approaches deliver more performant models, they do not show sufficiently strong generalization as the training only relies on the provided demonstrations.In this paper, we propose a self-refine Instruction-tuning method that allows for Smaller Language Models to self-improve their reasoning abilities.Our approach is based on a two-stage process, where reasoning abilities are first transferred between LLMs and Small Language Models (SLMs) via Instruction-tuning on synthetic demonstrations provided by LLMs, and then the instructed models self-improve through preference optimization strategies.In particular, the second phase operates refinement heuristics based on Direct Preference Optimization, where the SLMs are prompted to deliver a series of reasoning paths by automatically sampling the generated responses and providing rewards using ground truths from the LLMs.Results obtained on commonsense and math reasoning tasks show that this approach consistently outperforms Instructiontuning in both in-domain and out-domain scenarios, aligning the reasoning abilities of smaller and larger language models.