Analysis of Layer-Wise Training in Direct Speech to Speech Translation Using BI-LSTM
Lalaram Arya, Ayush Agarwal, Jagabandhu Mishra, S. R. Mahadeva Prasanna · 2022
Speech-to-speech translation (S2ST) is the process of translation of speech from one language to another. Traditional S2ST systems follow a cascaded approach, where three modules automatics speech recognition (ASR), machine translation (MT), and text-to-speech translation (TTS) are concatenated to obtain the final translated speech utterance. The cascaded nature of the system results in the propagation of errors from one module to another. This, in turn, leads to the degradation in the overall performance of the S2ST task. With the evolution of the deep learning approaches to speech processing, many attempts have been made to perform end-to-end and direct speech-to-speech translation (DS2ST). But most of these approaches rely on language transcripts in one way or the other. In this work, we aim to perform the DS2ST task without using language transcripts. In this direction we have performed three experiments: First, we have investigated the direct learning of mapping function from source to target language with the increase in the number of utterances. Second, we have analyzed how the learning function improves with an increase in the number of Bi-LSTM layers. Third, we have observed how the system behaves with the unknown speakers (not used during training) during inference. From the experiments, it has been observed that with the increase in the number of utterances and layers, the quality of translation improves. And also, with a speaker and text-dependent training of approximately 4.4 hrs of speech, the model can generate the target language utterance even for unknown speakers. Though the generated utterance quality is not that good, but intelligent to some extent to be perceived.