SINA microblog data retrieval
Yun Liu · Journal of Tsinghua University(Science and Technology) · 2011
With the large increase of SINA Microblog web users,analyses SINA Microblog data has aroused wide public concern.The data retrieval is a primary need for all these studies.This paper presents two data retrieval techniques based on opened APIs and web crawlers with webpage extraction.A program controls the methods and frequencies of the API queries for efficient data downloading via parsing of JSON objects.A traditional web crawler integrated with webpage extraction overcomes the shortcomings of opened SINA API interfaces that limit retrieval of the entire dataset without quantity and frequency limitations.Test results show that a combination of these two techniques allows complete data retrieval of SINA Microblog data.