Too big to mail: On the way to publish large-scale mobile analytics data
Ella Peltonen, Eemil Lagerspetz, Petteri Nurmi, Sasu Tarkoma · 2016
The Carat project started in 2012 has collected over 1.5 TB of data from over 850,000 mobile users all over the world. The project uses Apache Thrift to transmit data, and Apache Spark to run data analysis tasks, and the gist of the Carat analysis method has been published. While the Carat application code is open source, the data is much harder to share because of its size and privacy concerns. This paper outlines the challenges in sharing such a large-scale dataset with detailed information about smart devices, applications, and their users, and presents some solutions to these challenges.