To Comprehend the New: On Measuring the Freshness of a Document
Tirthankar Ghosal, Abhishek Shukla, Asif Ekbal, Pushpak Bhattacharyya · 2019
Detecting the novelty or freshness of an entire document is essential in this age of data duplication and semantic-level redundancy all across the web. Current techniques for the problem mostly root on handcrafted similarity and divergence based measures to classify a document as novel or non-novel. However, document-level novelty detection is relatively less explored in literature if compared to its sentence-level counterpart. In this work, we present a deep neural architecture to automatically predict the amount of new information contained in a document in the form of a novelty score. Along with, we offer a dataset of more than 7500 documents, annotated at the sentence-level to facilitate further research. Our approach which learns the notion of novelty and redundancy only from the data achieves significant performance improvement over the existing methods and adopted baselines (~17% error reduction). Also, our approach complies with the Two-Stage theory of human recall essential to comprehend new information.