RippleCOT: Amplifying Ripple Effect of Knowledge Editing in Language Models via Chain-of-Thought In-Context Learning
Zihao Zhao, Yuchen Yang, Yijiang Li, Yinzhi Cao · 2024
The ripple effect poses a significant challenge in knowledge editing for large language models.Namely, when a single fact is edited, the model struggles to accurately update the related facts in a sequence, which is evaluated by multi-hop questions linked to a chain of related facts.Recent strategies have moved away from traditional parameter updates to more flexible, less computation-intensive methods, proven to be more effective in the ripple effect.In-context learning (ICL) editing uses a simple demonstration Imagine that + new fact to guide LLMs, but struggles with complex multi-hop questions as the new fact alone fails to specify the chain of facts involved in such scenarios.Besides, memory-based editing maintains additional storage for all edits and related facts, requiring continuous updates to stay effective.As a result of the design limitations, the challenge remains, with the highest accuracy being only 33.8% on the MQUAKE-CF benchmarks for Vicuna-7B.To address this, we propose RIPPLECOT, a novel ICL editing approach integrating Chain-of-Thought (COT) reasoning.RIPPLECOT structures demonstrations as {new fact, question, thought, answer}, incorporating a thought component to identify and decompose the multi-hop logic within questions.This approach effectively guides the model through complex multi-hop questions with chains of related facts.Comprehensive experiments demonstrate that RIPPLE-COT significantly outperforms the state-of-theart on the ripple effect, achieving accuracy gains ranging from 7.8% to 87.1%.