A Multimodal Approach to Multispeaker Summarization and Mind Mapping for Audio Data
Hridayraj Modi, Aunali Patel, Isha Joshi, Pratik Kanani · 2023
With recent global events resulting in a shift in the work setting and way of life worldwide, most educational institutes and corporate organizations have been forced to conduct their day-to-day operations online. Lectures and meetings that are conducted online can be downloaded and users can store these recorded meetings. Owing to the latest technological advancements, storage capacities are no longer a hindrance when storing and using this data. However, storing these recorded meetings will only be helpful when users can access and browse them quickly. This paper proposes implementing a multi-speaker speech summarization model to benefit from the content captured in meeting recordings. Browsing through audio-visual data for information can be a time-consuming task for humans since manual summary generation would require a person to sit through the whole meeting. This can be overcome by combining Natural Language Processing techniques with text summarization to generate the transcript of a meeting. By performing speaker diarization, the various speakers in a multi-speaker meeting are identified, and accordingly, labels are assigned to every participant in the transcript. The model then performs summarization for individual speakers as well as the whole meeting and creates a mind map to represent the meeting minutes visually. Thus, this paper provides an efficient manner of reviewing meetings and labelling them for future reference.