Internet measurement data management challenges
Marina Fomenkov, kc claffy · 2011
Effective Internet measurement raises daunting issues for the research community and funding agencies. Improved understanding of the structure and dynamics of Internet topology, routing, workload, performance, and vulnerabilities remains disturbingly elusive, in part for lack of realistic and representative datasets available to scientific researchers. The dearth is understandable; measurement of operational Internet infrastructure involves managing more complex and interconnected dimensions than measurement in most scientific disciplines: logistical, financial, methodological, technical, legal, and ethical. CAIDA has been navigating these challenges with modest success for fifteen years, collecting, coordinating, curating, and sharing data sets for the Internet research and operational community in support of Internet science. Our three current biggest challenges which we hope to explore at the workshop are: sustainable collection, curation, and storage of large volumes of data; privacy-respecting sharing; and long-term archiving for reproducibility. 2. Overview of CAIDA data A complete list of the data that we currently collect and offer to the research community is available at [1]. It includes both ongoing data collections, data sets covering a single event, and archived collections. As of 2011, CAIDA captures and curate datasets from three primary sources: (i) macroscopic topology data (both IPv4 and IPv6) with the Archipelago infrastructure; (ii) traffic traces at Internet core backbone links; and (iii) traffic traces from the UCSD Network Telescope. Table 1 shows the amount of data resulting from our ongoing data collection operations in 2010. 2.1 Active measurement data Funded by NSF and DHS, Archipelago (Ark) is CAIDA’s newest active measurement infrastructure. As of June 2010 it