Analysis of Impact Between Data Analysis Performance and Database
Min Kyoung-ju, Jeongyun Cho, Manho Jung, Hyangbae Lee · Journal of information and communication convergence engineering · 2023
Engineering or humanities data are stored in databases and are often used for search services.While the latest deep-learning technologies, such like BART and BERT, are utilized for data analysis, humanities data still rely on traditional databases.Representative analysis methods include n-gram and lexical statistical extraction.However, when using a database, performance limitation is often imposed on the result calculations.This study presents an experimental process using MariaDB on a PC, which is easily accessible in a laboratory, to analyze the impact of the database on data analysis performance.The findings highlight the fact that the database becomes a bottleneck when analyzing large-scale text data, particularly over hundreds of thousands of records.To address this issue, a method was proposed to provide real-time humanities data analysis web services by leveraging the open source database, with a focus on the Seungjeongwon-Ilgy, one of the largest datasets in the humanities fields.