Learning to extract text-based information from the World Wide Web
Stephen Soderland · 1997
There is a wealth of information to be mined from narrative text on the World Wide Web. Unfortunately, standard natural language processing (NLP) extraction techniques expect full, grammatical sentences, and perform poorly on the choppy sentence fragments that are often found on web pages. This paper 1 introduces Webfoot, a preprocessor that parses web pages into logically coherent segments based on page layout cues. Output from Webfoot is then passed on to CRYSTAL, an NLP system that learns text extraction rules from example. Webfoot and CRYSTAL transform the text into a formal representation that is equivalent to relational database entries. This is a necessary first step for knowledge discovery and other automated analysis of free text. Information Extraction from the Web The World Wide Web contains a wealth of text information in the form of free text. Until a text extraction system transforms it into an unambiguous format, much of this information remains inaccessible to autom...