Installing OpenAI Environments on Windows 10

I had heard about OpenAI before actually using it from reading about general ML and AI topics. Mostly, because it was founded by Elon Musk. During my most recent grad class I was forced to use it to solve a project. This was one of the more frustrating assignments I have ever had. But, like most things like this, the most rewarding when I finally solved it.

The first thing to come up was that OpenAI environments (gyms) are not supported on Windows. Well, since I spend my day on Windows working in Visual Studio I decided to try and force my way through and get everything running. For the most part it worked, there are some big limitations but you can still interact with the environment. Of course, I could have created a Docker image and run in that but what fun would that be?

Here are my trails and tribulations

Installing Visual Studio 2017 with the Python Plugin:
This is a simple install/modify of Visual Studio. [Link]

Installing Anaconda:
This is also pretty straight forward. I had to download Anaconda 4.4 [Link] for Python 3.6.

Installing the Gyms:
This is the part where things get hairy. Since most of these items are NOT designed or supported in Windows. Because of this, you have to manually install some things and accept others just won’t work.

From the Anaconda Prompt:

git clone
cd gym
pip install -e .
pip install gym
pip install gym[all]

This will download all of the gyms but you WILL see errors. The biggest is that you can’t run ‘make’ on Windows for each of the Atari gyms. This is something I want to try and figure out.

You will need to download swig.exe [Link] and add it to your environment variables to build the Box2D items (I had to solve the Lunar Lander).

Once these are done, you can run a test application to ensure things are working.

import gym
env = gym.make('CartPole-v0')
for _ in range(1000):
 env.step(env.action_space.sample()) # take a random action

Box2D doesn’t work for Windows so I went to [Link] to download the wheel file. I then took that wheel file and installed it using ‘pip install’

Some Knows Issues:
ffmpeg [Download] wouldn’t install for me even though I could run it from the command line. This kept me from recording the videos that were posted to OpenAI. It wasn’t a huge deal but was a pain. In order to make the ‘wrapper’ class work so that I could at least upload the results I had to set video_callable to false.


Atari gyms won’t work without the ‘make’ commands. There are active discussions on the GitHub repo for OpenAI about getting them to work.