Advancing Moving Target Strategy with Bio-Inspired Reinforcement Learning to Secure Misconfigured Software Applications

Abstract

Misconfigurations in software systems are a persistent source of security vulnerabilities, particularly within static architectures that fail to adapt over time. Moving Target Defense (MTD) offers a proactive approach by dynamically altering the system’s attack surface, thereby reducing exposure. This paper builds upon an MTD model, RL-MTD, which leverages Reinforcement Learning (RL) to generate adaptive secure configurations. Although effective, RL-MTD faces limitations due to an unoptimized and sparse search space. To address this, two hybrid models—GA-RL and PSO-RL—are proposed, integrating Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) into the RL-MTD framework. Experiments on four misconfigured SUTs show both models outperform the baseline. Notably, PSO-RL yields the most secure configurations in most scenarios. The authors present a prototype demonstrating how PSO-RL could be applied on a constrained Windows 10 system to defend against an attack. These findings enhance MTDbased adaptive cybersecurity via optimized search.