Interactions with AI Systems: Trust and Transparency
Aaron Hunter · 2023
Modern AI systems are trained using sophisticated machine learning algorithms based on large data sets from a variety of sources. However, users that query these systems for information often have little knowledge about how they were trained or what information was used for training. As a result, users may believe the answers they are given to queries even in cases where they would not have trusted the data that was used to train the system. In this paper, we argue that trust in AI systems therefore relies heavily on transparency around the sources and methods used for training. In order to make this point precise, we introduce a model of a source network along with formal belief change operators that indicate how a user's beliefs should change when a trained system provides information. Using this formal framework, we demonstrate that there are cases where an agent can be deceived into believing information provided by an AI system, even if they would not have believed the information if it came directly from the sources used for training. We also show that our formal framework can be used to precisely state desirable properties for AI systems, which will guarantee that the system is only trusted when the underlying sources are trusted. Ethical considerations are discussed, highlighting the problems that occur when systems are allowed to obscure either the algorithms or the training data used.