Multi-Agent Arcade Learning Environment

The Multi-Agent Arcade Learning Environments are a multi-agent extension to the Arcade Learning Environment, perhaps the most important environments in Reinforcement Learning. The original environments were used to develop many of the innovations in Deep Reinforcement Learning (DRL), including the Deep Q Network (DQN), the first DRL algorithm. We added multi-agent support to these environments out of the hope that these environments could prove a reliable testing ground for multi-agent algorithm development.

• Code at https://github.com/Farama-Foundation/Multi-Agent-ALE
• Publication on arxiv
• A nice display of the environments is in the pettingzoo documentation.
• Code to train environments code using RLLIB and using ALL.

The core code is written in C++ (written by myself). A low level python interface is included, and high level python interface and documentation is available through PettingZoo.

Training efforts

We spent significant time attempting to learn effective policies in these environments via self-play. As discussed in the paper, our first attempt used agent indication and RLLib to train the environments (the bulk of the work was done by Luis Santos, I helped evaluate the trained models). Below is one of the more successful examples (on boxing).

RLlib posed various challenges and was unnecessarily complicated to work with, so we went and helped add multi-agent support to the Autonomous Learning Library, a well designed RL framework developed by Chris Nota. This resulted in much simpler, more reliable, and more scalable (for multiple experiments) code.

Some implementations using this multi-agent support for ALL are here and here.