Time-Stamped Language Model: Teaching Language Models to Understand The Flow of Events
Hossein Rajaby Faghihi, Parisa Kordjamshidi · 2021
Tracking entities throughout a procedure described in a text is challenging due to the dynamic nature of the world described in the process.Firstly, we propose to formulate this task as a question answering problem.This enables us to use pre-trained transformer-based language models on other QA benchmarks by adapting those to the procedural text understanding.Secondly, since the transformerbased language models cannot encode the flow of events by themselves, we propose a Time-Stamped Language Model (TSLM model) to encode event information in LMs architecture by introducing the timestamp encoding.Our model evaluated on the Propara dataset shows improvements on the published stateof-the-art results with a 3.1% increase in F1 score.Moreover, our model yields better results on the location prediction task on the NPN-Cooking dataset.This result indicates that our approach is effective for procedural text understanding in general.