Resource monitoring in information extraction
Jochen L. Leidner · 2007
It is often argued that in information extraction (IE), certain machine learning (ML) approaches save development time over others, or that certain ML methods (e.g. Active Learning) require less training data than others, thus saving development cost. However, such development cost claims are not normally backed up by controlled studies which show that such development cost savings actually occur. This situation in Language Engineering is contrasted with Software Engineering in general, where a lot of studies investigating system development cost have been carried out. We argue for the need of controlled studies that measure actual system development time in language engineering. To this end, we carry out an experiment in resource monitoring for an IE task: three named entity taggers for the same "surprise" domain are developed in parallel, using competing methods. Their human development time is accounted forusing a logging facility.We report development cost results for parallel implementations of a named entity tagger and present a breakdown of the development time for the three alternative methods. We are not aware of detailed previous parallel studies that detail how system development time is spent when creating a named entity tagger.