SITUATION-AWARE SPOKEN LANGUAGE PROCESSING
DK ROY · 2023
this paper is the construction of speech processing systems which leverage this third type of context. Our goal is to create situation-aware speech systems which integrate knowledge of relevant aspects of the current state of the world into spoken language learning, understanding and generation. Awareness of the situation is achieved by extracting salient information about the speaker's world from sensors including cameras, touch sensors, and microphones. A key challenge in this approach is to design and integrate linguistic and non-linguistic representations. This paper presents an implemented system which learns to understand and generate spoken utterances grounded in camera-derived visual semantics [18, 19]. Current work on a more sophisticated cross-modal language learning system is also introduced.