Increasing Convergence Rate in Decentralized Q-Learning for Traffic Networks
Sadredin Hokmi, ءMohammad Haeri · 2024
Given the importance of transportation, the speed of delivering goods and services to their destinations, and the need to prevent congestion in urban traffic networks, the use of effective tools to address these issues is necessary. In this paper, we resolve this problem within the framework of multi-agent reinforcement learning, where traffic networks are mathematically modeled as a congestion game. Decentralized multi-agent reinforcement learning shows promise for real-world cooperative tasks where agents lack access to global information, such as the actions of others. While independent Q-learning is frequently employed for decentralized training, the simultaneous policy updates by other agents result in non-stationary transition probabilities, leading to unreliable convergence. In addition to providing a solution to address non-stationarity for the given problem, we present an algorithm to increase the convergence rate by modifying the Q-values to reach the convergent value in fewer iterations. The proposed algorithm does not involve adjusting coefficients or changing the Q-update mechanism to enhance speed and prevent unwanted, endless wait loops, and it can be classified under a different and newer category. The simulation results demonstrate the algorithm’s effectiveness.