Correlation and Visualization Analysis of Large Scale Dataset GDELT
Fengcai Qiao, Kedi Chen · 2016
The Global Data on Events, Location, and Tone (GDELT) is a real time large scale database of global human society for open research which monitors the worlds broadcast, print, and web news since 1979, creating a free open platform for computing on the entire world. In this paper, first, we designed and implemented a data crawler, which collects metadata of GDELT database in real time and stores them in Hadoop Distributed File System (HDFS). Then, we proposed a hashbased method to correlate "Event" table, "Mentions" table and "GKG" table in GDELT, in order to digest every detailed information of each event. Finally, we took South Korea as example to make spatiotemporal visualization analysis, such as event spatiotemporal heat map, distribution of media attention and event extraction confidence dot map.