Improving Sequential Model Editing with Fact Retrieval
Xiaoqi Han, Ru Li, Hongye Tan, Yuanlong Wang, Qinghua Chai, Jeff Z. Pan · 2023
The task of sequential model editing is to fix erroneous knowledge in Pre-trained Language Models (PLMs) efficiently, precisely and continuously.Although existing methods can deal with a small number of modifications, these methods experience a performance decline or require additional annotated data, when the number of edits increases.In this paper, we propose a Retrieval Augmented Sequential Model Editing framework (RASE) that leverages factual information to enhance editing generalization and to guide the identification of edits by retrieving related facts from the fact-patch memory we constructed.Our main findings are: (i) State-ofthe-art models can hardly correct massive mistakes stably and efficiently; (ii) Even if we scale up to thousands of edits, RASE can significantly enhance editing generalization and maintain consistent performance and efficiency; (iii) RASE can edit large-scale PLMs and increase the performance of different editors.Moreover, it can integrate with ChatGPT and further improve performance.