Global Web Connectivity Block-page Detection Using Machine Learning and Deep Learning Approaches
Calle, Paola · Smith ScholarWorks (Smith College) · 2025
Block pages are web pages delivered in place of expected content, ranging from blank screens or generic error messages to detailed notices explaining access restrictions. While they are not the primary mechanism of censorship, block pages serve as valuable signals that, when systematically detected, can reveal critical insights into how and why Internet censorship is implemented. Traditional heuristic techniques—such as fingerprint matching, word frequency analysis, response length comparisons, and manual verification—are often labor-intensive, brittle, and difficult to scale. As web content becomes increasingly dynamic and region-specific, these methods quickly become outdated, especially in global detection contexts. This thesis introduces a scalable and adaptive detection pipeline that leverages machine learning (ML) and deep learning (DL) techniques to automate the identification of block pages and discover emerging censorship fingerprints. Drawing on up-to-date measurement data from the Open Observatory of Network Interference (OONI), our system is designed to adapt to the evolving nature of censorship practices across diverse networks and geographies. Our best-performing model achieved an accuracy of 96%, demonstrating strong potential for real-world deployment. We evaluate the pipeline using two weeks of OONI data, capturing temporal, geographic, and network-level diversity in web connectivity. Our key contributions include: (1) integrating state-of-the-art ML and DL models for adaptive block page detection; (2) uncovering previously undocumented censorship fingerprints; and (3) offering practical insights into the effectiveness and limitations of automated detection methods. To support continued research and transparency in this domain, we will release our models, detection pipeline, and source code via a public GitHub repository. By advancing the automation of censorship detection, this work contributes to global efforts to safeguard Internet freedom and uphold digital rights.