The NTCIR Workshop : the First Evaluation Workshop on Japanese Text Retrieval and Cross-Lingual Information Retrieval
Noriko Kando, Kazuko Kuriyama, 俊比古 野末, Koji Eguchi, Hiroyuki Kato, Soichiro Hidaka, Jun Adachi · 1999
This paper introduces the outline of the first NTCIR Workshop, which is the first evaluation workshop designed to enhance research in Japanese text retrieval and cross-lingual information retrieval. The test collection used in the Workshop consists of more than 330,000 documents with more than half are EnglishJapanese paired. Twenty-three groups from four countries have conducted IR tasks and submitted the search results. Various approaches were tested and reported at the Workshop. Finally some thoughts on the future directions of the NTCIR Workshop and evaluation of cross-lingual information retrieval with Asian languages are suggested. 1. Background and Aims The First NTCIR Workshop was held on August 30-Septebmer 1, 1999, in Tokyo[1]. The participation to the Workshop was limited to the active participants, i.e. the members of the research groups that submitted the results of the tasks, advisors and members of the organizing group. Many interesting papers with various approaches were presented at the Workshop and it ended in great enthusiasm. The third day of the Workshop was organized as the NTCIR/IREX Joint Workshop. IREX Workshop, the another evaluation workshop of IR and information extraction (named entity) using Japanese newspaper articles were held consecutively. The NTCIR Workshop was planed as part of the NTCIR project[2], which is intended to provide sound infrastructure to evaluate the search effectiveness of information retrieval systems with Japanese language and facilitate the IR research with Japanese language and cross-lingual retrieval including Japanese. The project is motivated by the recognition of the following situations: (1) Needs for a standard Japanese test collections (2) Need for cross-lingual retrieval (3) Need for the variety in text types (4) Need for the fundamental data for research into the intersection of IR and NLP The importance of the large-scale standard test collection in IR research are widely recognised. Stopping, stemming and query analysis are language depended procedures. Especially indexing texts written in Japanese or other East Asian languages like Chinese or Korean are quite different from those with English, French or other European languages since there is no explicit 1 This project is supported by Research for the Future Program JSPSRFTF96P00602 of the Japan Society for the Promotion of Science boundary (i.e. no space) between words in a sentence. Regarding other East Asian languages, there are large-scale test collections of Chinese and Korean. For Japanese, there is a standard test collection called BMIR-J2, consisting of 5,080 Japanese newspaper articles and ca.60 queries [3]. Although its contribution to Japanese IR research is tremendous, enhancement of the collection in both variety of text types and scale was needed. Cross-lingual retrieval is critical in the Internet environment. Moreover in the scientific texts, foreign language terms, sentences, or abstracts are often appeared in a Japanese text in their original spelling. Therefore cross-linguistic strategies are also critical for retrieval of Japanese scientific documents [4]. In order to respond the needs stated above, we aim to construct a large-scale test collection which is usable for cross-lingual retrieval and application of NLP to IR, and organize an evaluation workshop using it. The NTCIR Workshop has the following goals; (1) to encourage research in information retrieval, crosslingual information retrieval and related areas by providing a large-scale Japanese test collection and a common evaluation setting that allows cross-system comparisons (2) to provide a forum for research groups interested in comparing results and exchanging ideas or opinions in an informal atmosphere (3) to investigate effective methods for constructing largescale test collections and IR laboratory-type testing. The test collection used in the Workshop consists of more than 330,000 documents and more than half are English-Japanese paired. In the next section, we describe the tasks performed in the Workshop. Section 3 shows the test collection (NTCIR-1) used in the Workshop and section 4 introduces the evaluation results. The final section discusses some thoughts on future direction.