WIP: Automated Speech Proficiency Assessment for Conversations on Technical Subjects
Akhila Yaragoppa, Utkarsh Agarwal, Arnav Rustagi, Anushka Desai, SIDDHARTH SIDDHARTH, Brainerd Prince · 2024
This research-to-practice WIP paper presents a multimodal artificial intelligence (AI) system that assesses speech proficiency especially tuned for conversations on technical subjects particularly on the theme of AI. Critical thinking and clear communication skills are as important as technical career-specific skills. Automated speech proficiency tools only test for English speaking skills. However, the assessment of domain-specific technical communication skills is of utmost importance. The objective of this paper is to introduce an AI system for automated assessment of technical communication competency by providing an assessment for anyone desiring to converse on the theme of AI. The following three steps outline the approach followed in this paper. First, a multi-modal AI system has been designed by bringing together three pre-trained machine learning models each specializing in one domain related to speech proficiency estimation. Second, the multi-modal AI system is used to assess competence in proficiency of language communication in the context of technical conversations which particularly include: 1) content relevance 2) grammatical correctness, and 3) fluency of speech. Third, insights drawn from the multimodal system are compared to the speaker's self-perception of their proficiency. The uniqueness of our AI system is that it is able to combine the insights from these three models together towards a holistic speech proficiency analysis, especially for the technological domain. We validate the performance of our AI system under a controlled setting at a technological university. Our experiments led us to make the following discoveries about the use of our AI system to assess technical speech proficiency: 1) our AI system correctly analyzes that participants exhibit higher speech proficiency on topics in their domain of expertise, 2) grammatical correctness of speech remains unaffected when testing participants on topics with varied familiarity, and 3) participants' self-perception of their knowledge of various topics aligns with their ability to speak on the topic as measured by our AI system. We propose that our AI system would provide anyone interested in conversing proficiently on technological themes, especially AI, with the necessary tools to self-assess and track their progress.