Limits and Challenges of Embedding-based Question Answering in Export Control Expert System
Rafał Rzepka, Daiki Shirafuji, Akihiko Obayashi · Procedia Computer Science · 2021
In this paper, we report our initial findings from creating a question answering module for a dialog-based expert system which aims at advising users on export control regulations. We describe problems of data scarcity and knowledge transfer showing results of preliminary trials with utilizing contextual embeddings to extend keyword-based matching in our conversational expert system. We analyze pros and cons of the neural NLP techniques in systems which require precise information, for which no labelled data is available and which deal with less-resourced languages (Japanese in our case). Data scraped from governmental guidelines is presented and problems specific to legal documents in this area are discussed. Using examples from the preliminary experiments with deep learning-based methods we show when they fail and when they might be useful for further development of NLP systems operating on legal data. We also discuss evaluation-related problems and risks of imprecise output.