On the Validity of Internet Speed Test Tools and Broadband Measurement Programs
Jun Han, Minzhao Lyu, Vijay Sivaraman · 2022
Speed is etched into our psyche – consumers like to buy broadband or mobile service from the “fastest” provider; there is a whole industry of speed testing services; governments invest millions of dollars in national speed monitoring programs; and Internet Service Providers (ISPs) use speed test rankings in their advertising. Given how strongly speed shapes consumer, industry, and government behaviour, one would expect speed testing and comparison methodologies to be thoroughly studied and documented; however, this is not so. In this paper, we first highlight how test conditions skew results. Using popular speed test tools, we highlight some of the factors – test duration, number of threads, congestion control algorithm, and server locations – that significantly impact outcomes. Speed results should therefore only be interpreted in the strict context of their test conditions, and any generalisations should be discouraged and discarded. Our second contribution shows that national monitoring programs that rank ISPs on speed are largely flawed. By analysing public data from Australia and the U.K., we show that sampling imbalances across ISPs in terms of the test conditions (e.g., access technologies and locations) lead to biased results. Our study urges the community to restrain the emphasis placed on speed testing.