Schrödinger's Update: User Perceptions of Uncertainties in Proprietary Large Language Model Updates
Zilin Ma, Yiyang Mei, Krzysztof Z. Gajos, Ian Arawjo · 2024
Developers of proprietary large language models (LLMs), like OpenAI and Anthropic, often roll out updates, some announced and others silent, impacting transparency for users. We interviewed 21 LLM users, including end-users, developers, and academics, mainly using ChatGPT and GPT4. Many reported feelings of loss of control and uncertainty, and diminished trust due to silent updates and deprecations. Participants overall felt that proprietary LLMs were similar to traditional software with regards to updates, only differing in the impossibility in full transparency and explainability. Consequently, participants felt LLM companies departed from established software update norms without justification, in some cases appealing to the novelty of AI in order to excuse bad practices. As a result, users are inclined to shift towards fixed, open-source models, reserving proprietary LLMs for prototyping. We suggest strategies that LLM providers can adopt to better uphold transparency and trust during model updates and deprecations.