Speech To Image Translation Framework for Teacher-Student Learning
Vijaylaxmi Bittal, Anuradha Sanjay Bachhav, Makarand Shahade, Pradnya Rajendra Chavan, Bhavesh Anil Nikam, Ajinkya Anil Pawar · 2023
Most of the technical concepts written in textual way and these are communicated orally by teachers to students through limited images. Students not able to understand some of the complex topics in expected way. This will create a gap between subject and student's subject understandability. Representing and conveying through images is a convenient and effective way for Teaching Learning process. Filling this gap is very essential because understanding through visualization will enhance learning process. Though, the combo of speech encoding and computational translation in only one model causes a huge difficulty and many complexities. In order to reduce those efforts, speech to image translation framework plays significant role. With this motivation authors designed a framework for teachers-students learning process. In this framework, authors designed framework to abstract characteristics from audio files and trained model using unsupervised machine learning algorithm. In this firstly, a speech encoder encodes the audio and extract some significant and highlighted features from it and applying algorithm on this feature, Authors perform mapping process on these parameters and features of images which are present in dataset. After this process, GAN comes into play which produce corresponding image as output. In comparison to other methods like speech-text-image, text-image, speech-text models, our model approaches on synthesized dataset and reach to higher expectations of user.