Metadata management of terabyte datasets from an IP backbone network: experience and challenges
Sue B. Moon, Timothy Roscoe · 2001
INTRODUCTION Network measurements provide insight about real network traffic characteristics which cannot be obtained through modelling or simulation. Frequently, however, subsequent analysis of network data reveals the need for information previously discarded through sampling techniques or insufficient accuracy in measurement. The Sprint IP monitoring project began with a goal of acquiring enough data to answer most questions raised in the course of analysis. This is done by collecting information on every packet without any prior filtering or pre-processing of network traffic. In this paper we describe the systems issues of analysing data from the Sprint IP Monitoring Project, which collects very large sets of detailed packet-level data from a tier-1 backbone network. We report our early experiences managing these datasets and associated software within a small but growing research group. Based on our experience, we outline a comprehensive framework for efficiently managing t