A Method for Determining Utterance-Response Relationships Between Japanese Novel Lines for Constructing a Daily Dialogue Corpus
Kazuma Iwamoto, Kazuaki Ando · 2024
In recent years, with the development of various dialogue systems based on large language models, the demand for language resources has increased. However, there are not enough Japanese dialogue corpora for developing and evaluating chat-oriented dialogue systems. In addition, it is difficult to construct a corpus easily, because most of the conventional corpus construction methods are based on manual processes, and the construction cost is enormous. Therefore, we focus on the automatic construction method of a Japanese large chat-oriented daily dialogue corpus using novel lines. Various novels are produced every day, and they contain wide variety of lines. Since novels contain both consecutive and distant lines, the utterances and response relationship between lines must be considered to construct a dialogue corpus from extracted lines. In this paper, we propose a method for determining the utterance-response relationship between distant lines using a fine-tuned language model. From the evaluation results, we confirmed the validity of the proposed method and the feasibility of constructing a dialogue corpus from novels.