Automated Indonesian Essay Scoring and Holistic Feedback Using Bidirectional Encoder Representations for Transformers
Amalia Amalia, Maya Silvi Lydia, Rabiah Abdul Kadir, Faradhila Aulia Utami Tanjung, Dewi Sartika Br Ginting, Dani Gunawan · 2024
Developing an effective and accurate Automated Essay Scoring (AES) system for Indonesian essays involves overcoming several challenges, including limited datasets and issues related to grading efficiency and feedback generation. This study addresses these challenges by proposing an automated Indonesian essay grading and feedback system utilizing Bidirectional Encoder Representations from Transformers (BERT). The research employed a thorough process, including the translation of the ASAP dataset from the Kaggle competition titled “The Hewlett Foundation” into Indonesian, analysis and refinement of the dataset to include only relevant features, and fine-tuning using the pre-trained IndoBERT embeddings. To assess the relevance between the essay prompts and student responses, text similarity methods were applied outside the fine-tuning process. The final score is derived from the combination of the fine-tuned score and the relevance score. The model achieved a perfect accuracy of 1 and a strong kappa score of 0.82 during training. The alignment of the final score predictions with human feedback is validated by a Quadratic Weighted Kappa (QWK) score of 0.9, indicating a high level of agreement. This demonstrates that the automated scoring system effectively matches human judgment, highlighting its efficacy in evaluating essay quality.