A Study on Semantic Understanding of Large Language Models from the Perspective of Ambiguity Resolution
Shuguang Yang, F.R. Chen, Yiming Yang, Zude Zhu · 2023
Generative large language models (LLMs) have demonstrated outstanding performance in language understanding and generation tasks. Whether current LLMs can truly understand meaning or not is an open question. Semantic disambiguation is an important indicator to test machine language ability. To this end, three experiments were conducted to test the ambiguity resolution ability. In Experiment 1, the LLMs were asked to make a semantic discrimination on ambiguity phrases. The ChatGPT (May 3 Version) and ChatGPT Plus (May 3 Version) showed a random level performance of discrimination. To make the task easier for the LLMs, ambiguity sentences were presented with context in the Experiment 2. While the LLMs unified the context to fill the position for ambiguity word, the performance of semantic discrimination on these sentences remained at random level. In addition to test the performance of LMMs, prediction is one of the key ability of LLMs. The prediction by unifying the context is one of the key skill for LLMs. Therefore, the Experiment 3 further test the similarity between the human prediction mechanism and the strong generation capabilities of LLMs. Weak similarities were only found under the condition of local constraint features gradually accumulating, specifically, the cosine similarity of predicted sentence semantic vectors and ending word semantic vectors negatively correlated with close probability only. These series of experiments demonstrate that, it is far away to claim that LLMs truly understand human sentences.