HOW MUCH DO FORMAL NARRATIVE ANNOTATIONS DIFFER? A Proppian case study
Rens Bod · 2011
The formal study of narratives goes back to the Russian structuralist school, paradigmatically represented by the 1928 study Morphology of the Folktale by Vladimir Propp. Researchers in the field of computational narratology have developed the general Proppian methodology into various formal and computational frameworks for the analysis, automated understanding and generation of narratives. Methodological issues in this research field give rise to concrete research questions such as “How much does the representation of a narrative in a given formal framework depend on subjective decisions of the formalizer?'” touching philosophy of computing and philosophy of information. In order to approach the mentioned question, we consider the process of formal representation of a narrative as a natural analogue of the task of annotation in computational linguistics and corpus linguistics. We use the Russian folktales formalized by Propp and let them be formalized by annotators according to Propp's system, evaluating these results according to the standards of interannotator agreement. The formal study of narratives goes back to the Russian structuralist school, paradigmatically represented by the 1928 study Morphology of the Folktale by Vladimir Propp (1928) in which he identifies seven dramatis personae and 31 functions that allow him to formally analyse a corpus of Russian folktales. 2 R. BOD, B. LOWE, S. SARAF Researchers in the field of computational narratology (or “computational models of narrative”) have developed the general Proppian methodology into various formal and computational frameworks for the analysis, automated understanding and generation of narratives. Examples for this are Lehnert (1981)'s Plot Units, Rumelhart (1980)'s Story Grammars, Schank (1982)'s Thematic Organization Points (TOPs), Dyer (1983)'s Thematic Abstraction Units (TAUs), or Turner (1994)'s Planning Advice Themes (PATs). Over the last decades, the main interest of this research community lay in the technical challenges that the computational treatment of narratives brings, but recently, there is again increased interest in the methodological and conceptual issues involved, linking this research closely to questions of the philosophy of information (cf. the paper (Lowe to appear) presented at the 3rd Workshop for the Philosophy of Information). This interest is witnessed by workshops such as the recent AAAI workshop on Computational Models of Narrative that brought researchers from this field together with philosophers, narratologists and professional story tellers. The methodological issues involved give rise to concrete research questions such as 1. How do you compare formal frameworks of narrative? (Cf. Lowe 2010 and Lowe to appear.) 2. How do you assess the quality of a formal framework of narrative? 3. How much does the representation of a narrative in a given formal framework depend on subjective decisions of the formalizer? Question 1. is a genuinely philosophical question, but also the more technical questions 2. and 3. are very relevant for gaining philosophical insight into what constitutes the formal core of the concept of narrative. In this paper, we approach question 3. of the above list. To this end, we think of the process of formal representation of a narrative in a formal system as a natural analogue of the task of annotation in corpus linguistics and computational linguistics. Whereas typical annotation tasks involve annotation of sentences or discourses (cf., e.g., Marcus et al. 1993, Brants 2000, Passonneau et al. 2006), the formalization or annotation of a narrative is at the next level of complexity, involving sequences or systems of discourses, connected to a narrative. First studies suggest that question 3. is not easy to tackle for the following reasons: First, ambiguity which in typical linguistic annotation is a rather confined phenomenon becomes ubiquitous at the level of narratives: the natural answer to a formalization task is not one annotation, but a family of consistent annotations (cf. Lowe 2010, §2). Secondly, even allowing for multiple annotations, it is not clear whether consensus about whether a given annotation is a valid representation of a narrative is easy to achieve. Of course, these questions naturally reflect a well-known discussion from computational linguistics: in sentenceor discourse-level annotation, the quality of annotation is typically studied as inter-annotator agreement (Carletta et al. 1997, Marcu et al. 1999). For the annotation or formalization of narratives, no such analysis has ever been done, not even with the oldest and best-known formal approach to narrative structure, the Proppian narratemes. HOW MUCH DO FORMAL NARRATIVE ANNOTATIONS DIFFER? 3 In this study, we use English translations of the Afanas'ev tales formalized by Propp (Afanas'ev 1973), train a group of annotators in the use of Propp's system, and then let them formalize a selection of tales in that formal framework. We evaluate these results according to the standards of inter-annotator agreement from computational and corpus linguistics (Carletta et al. 1997).