PettingZoo

Motivation

Some of the most well known and inspiring advances in AI since the deep learning revolution has been in Deep Reinforcement Learning (DRL). Successes like Deepmind’s vivid learning of breakout and AlphaGo defeating world champion Go player Lee Sedol transformed the common understanding of what is possible to accomplish with computers, and helped drive the current deep learning revolution.

Recognizing the importance of DRL, OpenAI started an ambitious project to make DRL easily accessible to everyone. They designed gym, a standard library and API that focused on making environments easy to swap out, encouraging interchangeable RL code, and successfully making RL approachable to students and non-specialists, and also successfully made research and software development much easier for the specialized RL community.

Multi-agent reinforcement learning has gained massive popularity in recent years, with hundreds of papers being published in the area every year. Multi-agent tasks are important to the reinforcement learning community because cooperation and competition offer interesting challenges without having to increase the complexity of the environment. Reinforcement learning is also extremely exciting to the Multi-agent systems community, as they wish to study interactions between intelligent agents, and RL offers an automatic way for agents to gain intelligent behavior. However, the study of multi-agent reinforcement learning is much less accessible to students and non-specialists, and research and software development is much harder than ordinary reinforcement learning. We suspected this was due to the lack of a standard multi-agent API like Gym.

Overview

PettingZoo defines a powerful environment API which allows for reusable code for different types of environments. Its ambition is to have an equivalent role as OpenAI Gym in the multi-agent RL space. We made a number of unusual additions to our system to make it accessible to beginners in the field, such as extensive documentation, tutorials with popular RL learning frameworks, and more.

• Code at: https://github.com/Farama-Foundation/PettingZoo
• The website and documentation is at pettingzoo.farama.org
• Technical paper is at: arxiv
• Tutorial at towards data science

My contributions

• Influencing API design decisions, including the addition of the last() method, the agent_iter() method (see paper for details), and the inclusion of the parallel API.
• Implementing support for and writing documentation for MAgent, MPE, Atari, and Chess.
• Writing frontend code to implement website design, including code to automatically generate GIFs and AEC diagrams.
• General Maintenance, including responding to and fixing issues, maintaining environment versions.
• Rewriting and refactoring tests, maintaining CI testing.
• Writing developer documentation
• Helping write the academic paper

Advert

Here is a poster we showed at the NeurIPS Deep Reinforcement Learning workshop: