Multi-Agent Proximal Policy Optimization for Dynamic Multi-Channel URLLC Access

This work addresses the challenge of Dynamic Multi-Channel Access (DMCA) in the context of Ultra Reliable Low Latency Communications (URLLC), a framework subjected to notably stringent constraints, required by numerous Internet of Things (IoT) applications across various sectors. We introduce a theoretically grounded approach, leveraging Deep Multi-Agent Reinforcement Learning (MARL) to tackle this problem. While prior research has not fully addressed the DMCA problem in URLLC networks under time-varying heterogeneous channels and traffic profiles, nor provided robust theoretical guarantees in the multi-agent context, this paper adapts the recent theoretical framework of Trust Region Policy Optimization (TRPO) in MARL to meet the specific challenges and requirements of the URLLC-DMCA problem. Specifically, we introduce Multi Channel Access - Proximal Policy Optimization (MCA-PPO), a MARL algorithm that benefits from theoretical guarantees and effectively handles the partial observability and the combinatorial nature of the DMCA challenge. We validate the superiority of our proposed method across a variety of heterogeneous scenarios, in terms of traffic models and system parameters, and show that we outperform the traditional multiple access benchmark and learning algorithms.