Inter-indexer consistency studies, 1954-1975: a review of the literature and summary of study results
Lawrence E. Leonard · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · 1977
The English language is imprecise with semantic, syntactic, and syndetic structuring, which may easily cause a range of interpretation of many concepts, passages, phrases and words among different individuals.By the very nature of this imprecision, it is not surprising that inconsistency occurs among indexers and that there is no one right, correct, or exact assignment of index terms for any given document.Catalogers and indexers have long been aware of the problems associated with assignment of terms.They have attempted to minimize inconsistency by developing subject heading lists, authority files, thesauri, and rules for the use of these aids.Despite this conscious effort toward consistency, the literature on indexing does not reflect any attempt to measure the level of indexer consistency prior to study reo-rted in 1954 by Oliver Lilley.Since the mid-1950s, a number of studies have been conducted to measure the level of consistency of index term assignment by two or more indexers (interindexer consistency) and by an individual indexer upon reindexing a document (intra-indexer consistency).These studies use several different measures of indexer consistency.They vary considerably in research methodology and in the treatment of variables for the indexers, the documents selected, and the controls imposed on the indexing process.Not surprisingly, they also vary considerably in the level of indexer consistency reported.Inter-indexer consistency is a quantitative measure of the degree to which two or more indexers perceive the important information concepts contained in a document and represent these concepts using identical codes and/or terms.The raison d'etre for indexing documents is, however, to facilitate the retrieval of documents relevant to information queries made by users of the document base.The purpose of indexing is not necessarily to achieve 100 percent consistency among a group of indexers, but to retrieve documents that are relevant to search requests.This paper examines the inter-indexer consistency measures that have been used in studies of inter-indexer consistency, reviews criteria for index term matches, and discusses the major summaries that have been published on the topic.Inter-indexer consistency studies are then outlined, briefly sketching each study's methodology and reported results.An appendix summarizes all studies, categorizing each study under the following variables: indexer education, indexer experience, term type, indexing aids, document length, indexing depth, indexing time, indexing revision, indexing consistency, and relationship to retrieval effectiveness.Throughout this paper, these variables are defined as follows: Indexer education--undergraduate and graduate training Indexer experience--practical indexing/cataloging experience Term type--