Kim, SunghwanYoon, SeunghyunCho, Jin-HeeKim, Dong SeongMoore, Terrence. J. J.Free-Nelson, FredericaLim, Hyuk2023-04-172023-04-172022-12http://hdl.handle.net/10919/114524Reinforcement learning (RL) is a promising approach for intelligent agents to protect a given system under highly hostile environments. RL allows the agent to adaptively make sequential defense decisions based on the perceived current state of system security aiming to achieve the maximum defense performance in terms of fast, efficient, and automated detection, threat analysis, and response to the threat. In this paper, we propose a deep reinforcement learning (DRL)-based adaptive traffic inspection and moving target defense countermeasure framework, called 'DIVERGENCE,' for building a secure networked system. The DIVERGENCE provides two main security services: (1) a DRL-based network traffic inspection mechanism to achieve scalable and intensive network traffic visibility for rapid threat detection; and (2) an address shuffling-based moving target defense (MTD) technique to defend against threats as a proactive intrusion prevention mechanism. Through extensive simulations and experiments, we demonstrate that the DIVERGENCE successfully caught malicious traffic flows while significantly reducing the vulnerability of the network through MTD.application/pdfenCreative Commons Attribution-NonCommercial-NoDerivatives 4.0 InternationalInspectionIP networksSwitchesResource managementMonitoringUncertaintyControl systemsTraffic inspectionmoving target defensedeep reinforcement learningsoftware-defined networkingArticle - Refereedhttps://doi.org/10.1109/TNSM.2021.3139928194