What Happened
Zhang et al. introduced a novel framework in 2020 called the State Adversarial MDP (SA-MDP), which focuses on how attacks in reinforcement learning can vary based on the network used. They specifically argued that using the critic network (V(s)) would result in weaker attacks compared to those generated by the actor network (π(s)). This conclusion was backed by their empirical results derived from single-agent scenarios.
Why It Matters
The implications of these findings are significant for the field of adversarial reinforcement learning, particularly in multi-agent settings. If using the critic network indeed produces weaker attacks, it could influence how researchers and practitioners design their defenses against adversarial attacks. However, the conflicting observations in multi-agent scenarios raise questions about the generalizability of Zhang et al.'s claims.
Context
Adversarial reinforcement learning (ARL) is an emerging area where agents are trained to be robust against intentional perturbations in their environments. The SA-MDP framework proposed by Zhang et al. is particularly relevant as it provides insights into the mechanics of attacks within single-agent environments. However, the dynamics change significantly in multi-agent settings where interactions and competition between agents introduce additional complexities.
What It Means
The inconsistencies you've observed may stem from the inherent differences between single-agent and multi-agent environments. While Zhang et al.'s findings apply within a controlled single-agent framework, the dynamics of multi-agent systems, especially under Independent PPO (IPPO) and Graph Independent PPO (GPPO), can lead to different attack efficacy. It’s crucial to consider these contextual differences, as they may explain the disparity in your results. Further investigation into how multi-agent interactions impact adversarial strategies could provide deeper insights into the robustness of reinforcement learning techniques against attacks.
