Web Page Downloading and Classification
L.Q. Tran, Chul-Soo Moon, D.X. Le, Grid Thoma · 2002
Describes the processes of downloading and classifying Web-based articles in online medical journals as a preliminary step to extracting bibliographic data to populate MEDLINE/sup (R)/, the widely-used database of the National Library of Medicine (NLM). The processes are combined to develop an automated system named WPDC ("Web Page Downloading and Classification"). The system downloads the Web pages using Microsoft's Windows Internet API tool WinInet, and a combination of several artificial intelligence (AI) techniques, including the breadth-first search algorithm and the constraint satisfaction method. The breadth-first search algorithm and the constraint satisfaction method are then used to traverse the Web page's links, identify these pages as abstract, full text, PDF or image files, and recognize and generate the successors of the downloading pages.