Relational Dynamic Bayesian Networks with Locally Exchangeable Measures
Jaesik Choi, Kejia Hu · eScholarship (California Digital Library) · 2013
Handling large streaming data is essential for various applications such as network traffic analysis, social networks, energy cost trends, and environment modeling.However, it is in general intractable to store, compute, search and retrieve large streaming data.This paper addresses a fundamental issue, which is to reduce the size of large streaming data and still obtain accurate statistical analysis.As an example, when a high-speed network such as 100 Gbps network is monitored, the collected measurement data rapidly grows so that polynomial time algorithms (e.g., Gaussian processes) become intractable.One possible solution to reduce the storage of vast amounts of measured data is to store a random sample, such as one out of 1000 network packets.However, such static sampling methods (linear sampling) have drawbacks: (1) it is not scalable for high-rate streaming data, and ( 2) there is no guarantee of reflecting the underlying distribution.In this paper, we propose a dynamic sampling algorithm that reduces the storage of data records in exponential scale, and still provides accurate analysis of large streaming data.We also build an efficient Gaussian Process with the fewer samples.We apply this algorithm to large data transfers in high-speed networks, and show that the new algorithm significantly improves the efficiency of network traffic prediction.1