Kitten: a tool for normalizing HTML and extracting its textual content
Mathieu-Henri Falco, Véronique Moriceau, Anne Vilnat · 2012
The web is composed of a gigantic amount of documents that can be very useful for information extraction systems.Most of them are written in HTML and have to be rendered by an HTML engine in order to display the data they contain on a screen.HTML files thus mix both informational and rendering content.Our goal is to design a tool for informational content extraction.A linear extraction with only a basic filtering of rendering content would not be enough as objects such as lists and tables are linearly coded but need to be read in a non-linear way to be well interpreted.Besides these HTML pages are often incorrectly coded from an HTML point of view and use a segmentation of blocks based on blank space that cannot be transposed in a text file without confusing syntactic parsers.For this purpose, we propose the Kitten tool that first normalizes HTML files into unicode XHTML files, then extracts the informational content into a text file with a special processing for sentences, lists and tables.