Can Twitter API Be Bypassed? A New Methodology for Collecting Chronological Information Without Restrictions
Aldo Hernandez-Suarez, Gabriel Sánchez-Pérez, Karina Toscano-Medina, Rocío Toscano-Medina, Victor Martinez-Hernandez, Jesús Olivares-Mercado, Hector Manuel Perez-Meana, Victor F. Sanchez · Frontiers in artificial intelligence and applications · 2018
Retrieving information from social networks is a first and primordial step in many data analysis fields such as Natural Language Processing and Machine Learning. Important data science tasks rely on historical data gathering for further predictive results. Recent works use public platforms for collecting public streams of information like Twitter API, which allows querying chronological tweets from periods no longer than three weeks. In this paper, we present Twitter Scrapy, a new methodology for collecting historical tweets from time periods of arbitrary duration using web scraping techniques that bypass Twitter API restrictions.