Quote to Explain: Using Multimodal Metalinguistic Markers to Explain Large Language Models' Understanding Capabilities
Milena Belošević, Hendrik Buschmeier · 2024
Starting from the assumption that LLMs are systems bearing only formal but not functional linguistic competence, this short paper explores how the understanding capabilities of LLMs could be implicitly explained based on a “pause and reflect” strategy. Specifically, we propose to include a virtual embodied agent in human interactions with LLM-based chatbots. The agent will use air quotes as multimodal metalinguistic markers to explicitly point to those parts of the LLM’s output that are relevant to explaining the LLM’s meaning understanding capabilities. At the same time, by scaffolding users to perceive the output as ‘mentioned language’ inferred from a metalinguistic function of multimodal markers, the agent implicitly explains how the meaning of the output should be understood. In this proposal, users will actively participate in the co-construction of the implicit explanation by providing feedback and deciding when and to what extent the agent’s scaffold (e.g., the air quotes) is used.