iobes: Library for Span Level Processing
Brian Lester · 2020
Many tasks in natural language processing, such as named entity recognition and slotfilling, involve identifying and labeling specific spans of text.In order to leverage common models, these tasks are often recast as sequence labeling tasks.Each token is given a label and these labels are prefixed with special tokens such as B-or I-.After a model assigns labels to each token, these prefixes are used to group the tokens into spans.Properly parsing these annotations is critical for producing fair and comparable metrics; however, despite its importance, there is not an easy-to-use, standardized, programmatically integratable library to help work with span labeling.To remedy this, we introduce our opensource library, iobes.iobes is used for parsing, converting, and processing spans represented as token level decisions.