Choosing Scrapy
Daniel Myers, James W. McGuffee · 2015
Scrapy is an open-source web-crawling platform implemented in Python. As part of an undergraduate research project, the lead author created a web crawler with Scrapy that utilizes regular expressions to extract relevant data from geographically targeted websites. This student project is part of a larger, longer term research project to determine the viability of creating an interactive geographically constrained social services client matching system. This paper describes the reason Scrapy was chosen for this project and explores the viability of Scrapy as a powerful and robust tool for undergraduate projects in the computing sciences.