Chartlytics: Chart Image Classification and Data Extraction
Yasmin Al-Kady, Nurhan Abdel-Wahab, Farah Hassanein, Rana Al-Alfy, Lama El-Malatawy, Rowym Ammar, Mirna Al-Shetairy, Salsabil Amin · 2023
Chart images play a vital role in visualizing data, but automatically extracting the values from these charts poses a significant challenge. The diverse range of chart styles makes it difficult to employ pure rule-based data extraction methods effectively. Developing a system that generates textual descriptions of images would improve the accessibility of digital content for individuals with visual impairments. To address this issue, Chartlytics was developed, a user-friendly, accessible application that can create textual descriptions for a given chart image and display them for the user in the form of plain understandable text or a sound that people with complete loss of vision can easily hear. Our proposed framework has three main parts that work together in a sequence. First, an image is passed through a classification model to decide its chart type. Then, based on the chart type, the image is processed by a specific component detection model to identify the bounding boxes of chart components. Next, the image is fed into the data extraction module, where rule-based techniques are used to extract the relevant information from the chart. Finally, the output data is processed by a chart description generator that is based on a comprehensive predefined template for each chart type. Our solution achieves an accuracy of 99.42% in classifying the chart image using the MobileNet model, a 0.99 mAP50 score in detecting chart components using a YOLO model, and an accuracy that ranges between 89.2% and 98.5% in extracting data from charts according to each chart type.1 2