Using TWIG: India's past versus present via topic modeling
Newton McCollum, Andrew Fayed, Benjamin McIntosh, James Carignan, Deepti Joshi · Journal of computing sciences in colleges · 2016
In this research, we look at past data collected through a Java programmed web crawler which scrapes the archives of the Indian Express (a leading national newspaper from India) for articles from 2000 until present day, and retrieves them as text files which are then stored in the MongoDB database. In addition, using TWIG (The Web Information Gatherer), we look at current data collected through Twitter and online news sources with RSS Feeds. To this dataset of past and present, we apply topic detection algorithm --- LDA, that allows us to form a generalization of an article which can be used to conclude a general idea about a region during a given time period. The result is visualized as a word cloud for every three years, from 2000 until 2016. This enables us to observe the changes and trends in the topics of interest in India. Similar analysis can be done with countries all over the world. The applications are extremely relevant to the intelligence communities.