Pitfalls of data-driven networking
P. C. Sruthi, Sanjay Rao, Bruno Ribeiro · 2020
This paper motivates the need to support counterfactual reasoning (i.e., answer "what-if " questions about events that did not occur) when collecting network data. We focus on video streaming - e.g., given logs of a video session, a video publisher may ask whether a user would continue to experience no rebuffering events if the lowest quality video choice were eliminated. We discuss potential pitfalls related to counterfactual reasoning, and argue that dynamic network state (e.g., bandwidth) serves as a confounding yet hidden (latent) feature that complicates such analyses. We illustrate the challenges, and present preliminary methods to address them using concrete examples. Our evaluations show that existing approaches, including randomized trials (collecting data from an algorithm that selects bitrates randomly), are by themselves inadequate for counterfactual reasoning related to video streaming, and must be supplemented by techniques that explicitly infer latent features.