AoI Minimization Using Multi-Agent Proximal Policy Optimization in UAVs-Assisted Sensor Networks
Yousef Emami, Kai Li, Yong Jie Niu, Eduardo Tovar · 2023
Unmanned Aerial Vehicle (UAV) swarm can be employed to collect time-sensitive data of ground sensors in remote and hostile areas. Inadequate design of UAVs' trajectories and data collection schedule incur delay and negatively impact the information freshness of ground sensors. This paper aims to jointly optimize the trajectories and data collection schedules of multiple UAVs to minimize the average Age of Information (AoI), adapting to the AoI of the ground sensors, and the trajectories of the UAVs. The optimization is formulated as a multi-agent Markov decision process (MMDP), where network states consist of AoI at the ground sensors and the flight trajectories. In practice, a multi-UAV-assisted sensor network contains a large number of network states and actions in MMDP. Exploring the actions of multiple agents in a large state space results in considerable training uncertainties that destabilize the AoI minimization. For stabilizing the formulated MMDP, we propose an onboard Proximal Policy Optimization-based flight resource allocation scheme (PPO-FRAS), which conducts an on-policy learning to optimize the trajectories of the UAVs and data collection schedule of the ground sensors. Numerical results show that the proposed PPO-FRAS achieves 28% and 59% lower AoI than the existing trajectory planning solution based on Deep Q-Network and the greedy algorithm, respectively.