The Big Digger & Puzzler System for Harvesting & Analyzing Data from Social Networks
Einar J. Holsbø, Phuong Hoai Ha, Otto J. Anshus · 2013
The Big Digger & Puzzler is a distributed system for harvesting and analyzing data from social networks. The system is distributed on users’ PCs around the world. A Digger collects data as specified by the user; a Puzzler analyzes data collected by the Digger, and can contact other Puzzlers to request analytics data on-demand. Puzzlers don’t share the raw data with each other, only analytics: this cuts down on bandwidth- and storage requirements, and effectively distributes the compute workload across users interested in approximately the same topics and analytics. On a large enough scale, enough relevant analytics will be available to serve many different requests, statistically speaking.