Experience Constrained Hierarchical Federated Reinforcement Learning for Large-scale UAV Teams in Hazardous Environments
Conventional federated learning assumes that greater learner participation improves training performance, by leveraging abundant, independently generated local data. However, in federated reinforcement learning (FRL) for unmanned aerial vehicle (UAV) teams in hazardous environments where experience generation is severely constrained by safety considerations, energy limitations, and mission duration, this assumption may break. This work introduces Experience-Constrained Hierarchical Federated Reinforcement Learning (EC-HFRL), a framework in which clusters act as federated learning agents, while multiple intra-cluster learners represent parallel learning resources that reuse a shared experience pool. We show that increasing participation does not necessarily improve learning performance. Instead, learning performance is strongly associated with experience reuse strategy and the dominance of key analytically identified gradient transition experiences within a cluster. In particular, minibatch size primarily determines effective replay exposure, while higher intra-cluster participation increases reuse level. Empirical results demonstrate that the performance regimes are strongly associated with the structure of the learning signal, rather than federated aggregation effects, clarifying the limited and secondary role of learner participation in experience-constrained FRL.
Bio: Simon Khan is a research computer scientist at the Air Force Research Laboratory (AFRL) in Rome, New York, where his research centers on artificial intelligence for autonomous decision systems. With nearly 20 years of professional experience within the US government, Khan works on reinforcement learning, explainable AI, swarm intelligence, multi-agent cooperation, and human-AI collaboration, with additional interests in cybersecurity, space and command and control. He has led, worked and supported research on autonomous command-and-control systems, neuro-symbolic belief maps, reinforcement-learning safety, and explainable reinforcement learning, and served as a technical advisor for the Cyber Mission Assurance Technology program supporting U.S. Transportation Command. He has led and supported major interdisciplinary research programs and has advised more than 40 researchers, professors, and interns. Khan has organized international and professional research activities as a co-chair, including NATO AVT Platform AI-centric collaboration, and AFOSRernational activities (e.g., EOARD). An IEEE Senior Member, he has authored or co-authored over 50 publications. He holds a Ph.D. in electrical and computer engineering from Clarkson University, an M.S. in computer engineering from Syracuse University, an M.S. in information systems management from National University, and a B.S. in electrical engineering from Stony Brook University. In addition, he has received and nominated for multiple awards within the US government.