Function Approximation
Nimish Sanghi · Apress eBooks · 2021
In the previous three chapters, we looked at various approaches to planning and control, first using dynamic programming (DP), then using the Monte Carlo approach (MC), and finally using the temporal difference (TD) approach. In all these approaches, we always looked at problems where the state space and actions were both discrete. Only in the previous chapter toward the end did we talk about Q-learning in a continuous state space. We discretized the state values using an arbitrary approach and trained a learning model. In this chapter, we are going to extend that approach by talking about the theoretical foundations of approximation and how it impacts the setup for reinforcement learning. We will then look at the various approaches to approximating values, first with a linear approach that has a good theoretical foundation and then with a nonlinear approach specifically with neural networks. This aspect of combining deep learning with reinforcement learning is the most exciting development that has moved reinforcement learning algorithms to scale.