Markov Decision Processes with Information Costs
Stefanie Winkelmann · Universitätsbibliothek der FU Berlin Hochschulschriftenstelle u. Dokumentenserver · 2013
For several decades, the theory of Markov decision processes has been successfully used to model situations of controlled stochastic dynamics in various application areas. Beside the original setting which assumes the controlled process to be completely observable at all times, there exist several variants of Markov control theory for cases of incomplete state information. All these variants underlie special restrictions: The time scale of the process has to be discrete, the state space is assumed to exhibit a special ordered structure or the type of dynamics is in some sense predefined. In this thesis we develop a novel model of Markov control with incomplete state information which is applicable to all kinds of continuous-time dynamics on a discrete state space. The observation of the process and the choice of actions take place at discrete points in time which themselves are subject to the control of the decision maker. Each observation produces a fixed amount of information costs which are included in the considered cost criteria. The chosen action determines the stochastic dynamics of the process within the next time period of hidden progress. The resulting combined optimization of observation times and interaction is extensively studied in this thesis. Given the new setting, we redefine the two criteria of discounted costs and average costs in an appropriate way. Both criteria are analyzed subsequently. Their relation to the cost criteria considered in the original Markov control theory is established and the impact of the control parameters and of the information cost parameter is explored. One main result is the reformulation of the Bellman equation which delivers the basis for an efficient numerical calculation of the optimal control policy. The corresponding value function of optimal costs is discovered to be monotone and continuous with respect to the information cost parameter and to coincide with the value function of the original setting when considering vanishing information costs. The proposed model not only permits a coherent and productive theoretical analysis, but also forms the basis for an interesting real-world application. We consider the dynamics of HIV and use the developed theory to calculate optimal therapeutic strategies for resource-rich and resource-poor settings. We discover, among other things, that a decrease of diagnostic costs in resource- poor settings would significantly enhance the medical success of cost optimal therapies. This thesis provides a comprehensible framework for analyzing situations of controlled dynamics which are not permanently observable. The framework is based on the two fundamental assumptions that a state test always delivers instantaneous and perfect information and that the action can only be adapted after such a test. A question of interest is how far these assumptions can be eased in order to generalize the control model. Finding an answer to this question proves to be a topic of future research.