Is this the end? Two-step tokenization of sentence boundaries
Linda Wiechetek, Sjur Nørstebø Moshagen, Thomas Omma · 2019
A period does not only mark the end of a sentence; it can also be part of an abbreviation and numerical expressions.When analysing corpus text linguistically we need to know where a sentence begins and where it ends.In traditional corpus analysis, typically a sentence is identified before linguistic analysis is performed.In this work we propose an approach where we do basic linguistic analysis before we decide what a sentence is.As the interpretation of a period after abbreviations and numerical expressions is ambiguous, we leave the ambiguity in the initial tokenization.In a second step we remove the ambiguity based on the linguistic context of the period.We compare the previous approach to the new one and show how the new two-step approach to tokenization improves the identification of sentence boundaries.