Out of Sight, Out of Mind: Detecting Orphaned Web Pages at Internet-Scale
Stijn Pletinckx, Kevin Borgolte, Tobias Fiebig · 2021
Security misconfigurations and neglected updates commonly lead to systems being vulnerable. Especially in the context of websites, we often find pages that were forgotten, that is, they were left online after they served their purpose and never updated thereafter. In this paper, we introduce new methodology to detect such forgotten or orphaned web pages. We combine historic data from the Internet Archive with active measurements to identify pages no longer reachable via a path from the index page, yet stay accessible through their specific URL. We show the efficacy of our approach and the real-world relevance of orphaned web-pages by applying it to a sample of 100,000 domains from the Tranco Top 1M.