Experiences with TA-Bot in CS1
Jack Forden, Alexander Gebhard, Dennis Brylow · 2023
Automated Assessment Tools (AATs) have been used in undergraduate CS education for decades. TA-Bot, a modular AAT, has existed in some form for 25 years serving thousands of students across multiple universities. Class sizes throughout the last decade have continued to grow, while the number of instructors remains stagnant. AATs help instructors mitigate issues without additional resources, while simultaneously providing students with helpful feedback. The research team implemented novel features into the new, web-based TA-Bot such as dynamic rate limiting between submissions, custom code style feedback, and a gamified points system. The experiment discussed in this paper used TA-Bot over the course of three semesters involving 145 students in CS1. During the first semester, student and instructor feedback was collected on how to improve the tool. The second semester was used to rate limit submissions using a new dynamic rate limiting system. Finally, the third semester of TA-Bot was used as a control group with simple submission input/output checking. Instructors found that TA-Bot helped mitigate issues with continual increases in class sizes. When using TA-Bot with a dynamic rate limit, students were more inclined to start their assignment earlier. In addition to this, TA-Bot provides students with the ability to compare their solution against test cases, while simultaneously providing code-style advice using curated novice-friendly examples.