Sparse Activation Editing for Reliable Instruction Following in Narratives
Runcong Zhao, Chengyu Cao, Qinglin Zhu, Xiucheng Ly, Shun Shao, Lin Gui, Ruifeng Xu, Yulan He · 2025
Complex narrative contexts often challenge language models' ability to follow instructions, and existing benchmarks fail to capture these difficulties.To address this, we propose Concise-SAE, a training-free framework that improves instruction following by identifying and editing instruction-relevant neurons using only natural language instructions, without requiring labelled data.To thoroughly evaluate our method, we introduce FREEIN-STRUCT, a diverse and realistic benchmark of 1,212 examples that highlights the challenges of instruction following in narrative-rich settings.While initially motivated by complex narratives, Concise-SAE demonstrates stateof-the-art instruction adherence across varied tasks without compromising generation quality.