Unity ML Agents | Penguins

Updated Tutorial Available!

We’ve updated the original tutorial for the latest version of ML-Agents!

Check out all four parts of that tutorial here:

Reinforcement Learning Penguins (Part 1/4)
Reinforcement Learning Penguins (Part 2/4)
Reinforcement Learning Penguins (Part 3/4)
Reinforcement Learning Penguins (Part 4/4)

 

Original Tutorial

This is a companion post to the Unity ML Agents YouTube tutorial. You’ll learn how to use machine learning, specifically reinforcement learning in Unity to train penguins to find fish and feed their babies.

ML-Agents Code

In this video, I'm using Release 0.8.2, which you can get here:

https://github.com/Unity-Technologies/ml-agents/releases/tag/0.8.2

👉 The latest version has changed quite a bit so if you have trouble, you can always roll back to 0.8.2. Here are the changes you should know about.

⚠Version 0.9 Update

Version 0.9 was released on August 2nd, 2019 (three days after I posted this #facepalm) and requires a change to the format of trainer_config.yaml. I updated the code in the link Gumroad download link below. If you are using version 0.9 instead of 0.8.2, make sure to use trainer_config-0.9.yaml instead. (More details here)

⚠Version 0.11 Update

Version 0.11 was released at the beginning of November, 2019 and introduces a change with how agents and trained .nn models work. The brains are gone, which means:

  1. Agents now need a “BehaviorParameters” script attached, trained .nn files get hooked up here

  2. There’s no such thing as a “PlayerBrain”, this will need to be done with the Heuristic() function of the agent and the “Use Heuristic” checkbox

  3. The Academy broadcast hub is gone

Here’s a Heuristic function you can use (it goes in the PenguinAgent class)

/// <summary>
/// Read inputs from the keyboard and convert them to a list of actions.
/// This is called only when the player wants to control the agent and has set
/// Behavior Type to "Heuristic Only" in the Behavior Parameters inspector.
/// </summary>
/// <returns>A vectorAction array of floats that will be passed into <see cref="AgentAction(float[])"/></returns>
public override float[] Heuristic()
{
    float forwardAction = 0f;
    float turnAction = 0f;

    if (Input.GetKey(KeyCode.W))
    {
        // move forward
        forwardAction = 1f;
    }
    if (Input.GetKey(KeyCode.A))
    {
        // turn left
        turnAction = 1f;
    }
    else if (Input.GetKey(KeyCode.D))
    {
        // turn right
        turnAction = 2f;
    }

    // Put the actions into an array and return
    return new float[] { forwardAction, turnAction };
}

⚠Version 0.12 Update

Version 0.12 was released at the beginning of December, 2019 and there are a three small changes. You can read the official Migration Guide for more info.

1. Barracuda Package Dependency

ML-Agents version 0.12 takes a dependency on the "Barracuda" package in the Unity Package Manager. If you do not install this, ML-Agents will not work and you will get a lot of error messages.

To fix this:

  1. In the top menu, go to "Window" > "Package Manager"

  2. Click "Advanced" and choose "Show preview packages"

  3. Find and click on the "Barracuda" package

  4. Click "Install" and allow the installation to complete

2. Use Heuristic Checkbox Replaced with Behavior Type Dropdown

The Use Heuristic checkbox in Behavior Parameters has been replaced with a Behavior Type dropdown menu. This has the following options:

  • Default corresponds to the previous unchecked behavior, meaning that Agents will train if they connect to a python trainer, otherwise they will performance inference.

  • Heuristic Only means the Agent will always use the Heuristic() method. This corresponds to having "Use Heuristic" selected in 0.11.0.

  • Inference Only means the Agent will always perform inference.

3. AgentAction() Change

This one is easy. The AgentAction() override no longer needs the "textAction" parameter. We weren't using it, so it has no impact aside from needing to be removed.

In version 0.11, the code should be:

public override void AgentAction(float[] vectorAction, string textAction)

In version 0.12, the code should be:

public override void AgentAction(float[] vectorAction)



⚠Version 0.13 Update

They changed how Reset Parameters work in the Academy (in January 2020) so that you need to register a callback function for any time a curriculum value changes.

To accommodate for this, you can update your PenguinAcademy as follows.

public class PenguinAcademy : Academy
{
    /// <summary>
    /// Gets/sets the current fish speed
    /// </summary>
    public float FishSpeed { get; private set; }

    /// <summary>
    /// Gets/sets the current acceptable feed radius
    /// </summary>
    public float FeedRadius { get; private set; }

    /// <summary>
    /// Called when the academy first gets initialized
    /// </summary>
    public override void InitializeAcademy()
    {
        FishSpeed = 0f;
        FeedRadius = 0f;

        // Set up code to be called every time the fish_speed parameter changes 
        // during curriculum learning
        FloatProperties.RegisterCallback("fish_speed", f =>
        {
            FishSpeed = f;
        });

        // Set up code to be called every time the feed_radius parameter changes 
        // during curriculum learning
        FloatProperties.RegisterCallback("feed_radius", f =>
        {
            FeedRadius = f;
        });
    }
}

Then, instead of penguinAcademy.resetParameters, you would use these academy accessors as follows.

SpawnFish(4, penguinAcademy.FishSpeed);

and

feedRadius = penguinAcademy.FeedRadius;

Newer Version?

Check out the official Migration Guide.

Downloads

Code and 3D Meshes (Free)

https://gumroad.com/l/TZNBD 👈 $0+ on Gumroad means you can get them *completely for free* by typing in a 0, no account required, just an email address.




Setup

As mentioned in the video, the first thing you'll need to do is follow the Installation instructions and the Basic Guide. These will get you up and running with Unity ML-Agents and ready to start the tutorial.

https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md

https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Basic-Guide.md

I highly encourage you to check out the ML-Agents Documentation, which will help you understand the project in more depth and provide a place to discuss issues that are not specific to the pig project.

https://github.com/Unity-Technologies/ml-agents/tree/master/docs

Helpful Additions

Training Command

The command I used to train (in my Anaconda Prompt) with curriculum was:

    mlagents-learn config/trainer_config.yaml --curriculum=config/curricula/penguin/ --run-id=penguin_01 --train

If you are using version 0.9, you’ll have to point to the correct trainer_config:

    mlagents-learn config/trainer_config-0.9.yaml --curriculum=config/curricula/penguin/ --run-id=penguin_01 --train

Ray Perception Sensor Component

Currently the tutorial uses RayPerception3D components, but these are now deprecated as of version 0.12. If you’re interested in how to use the new RayPerceptionSensorComponent3D, check out our tutorial.

AI Flight with Unity ML-Agents

You know what’s a bummer? YouTube videos can’t be updated. If you want an up-to-date, end to end, tutorial, we created a very affordable AI Flight course on Udemy. It’ll teach you how to create a complete airplane racing game with ML-Agents! On top of that, by paying a little bit for it, you’ll support our future work. ❤

Unity, 3D, Deep LearningAdam KellyMachine Learning, Unity ML Agents6 Comments