How Google uses Reinforcement Learning to Train AI Agents in the Most Popular Sport in the World

That was the strategy followed by the Google Brain team.

  The idea of applying reinforcement learning to football environments seems intuitive.

After all, reinforcement learning has been behind some of the biggest breakthroughs in AI from the creation of AlphaGo to surpassing humans in complex multiplayer environments such as Dota2 or Quake III.

Reinforcement learning provides a model in which AI agents can master the rules of an environment by trial and error instead of predefined training datasets.

In general, games provide a great environment for reinforcement learning agents as they test new ideas in a reproducible manner so the idea of applying those principles to football seem intuitive.

However, creating a reinforcement learning for football is far from trivial and it comes with a very unique set of challenges:Those are some of the key reasons why football has managed to elude most AI algorithms.

The Google Brain team balanced those challenges with state of the art reinforcement learning models that help master football in a very unique way.

  The Google Research Football project is a reinforcement learning environment in which agents learn to play football by simply playing.

The current version of the platform is based on three fundamental components:Football EngineThe Football Engine is an advanced football simulation based on the popular Gameplay Football environment.

The engine simulates a complete football game, and it accepts input actions from both teams which includes the most common football aspects, such as goals, fouls, corners, penalty kicks, or offsides.

From the reinforcement learning standpoint, the Football Engine includes a series of relevant properties that are worth highlighting:The current version of the Football Engine was written in C++, allowing it to be run on off-the-shelf machines, both with GPU and without GPU-based rendering enabled.

This allows it to reach a performance of approximately 25 million steps per day on a single hexa-core machine.

Football BenchmarksThe Football Engine provides the fundamental building blocks for researchers to try new ideas for mastering football.

However, we still need a well-establish mechanism for objectively evaluating the viability of those ideas.

The Football Benchmark evaluates different strategies against a predefined set of tasks.

Functionally, the goal in these benchmarks is to play a “standard” game of football against a fixed rule-based opponent that was hand-engineered for this purpose.

The current version of Football Benchmark provides three versions: the Football Easy Benchmark, the Football Medium Benchmark, and the Football Hard Benchmark, which only differ in the strength of the opponent.

The Google Brain team tested Football Benchmarks with two states of the art reinforcement learning algorithms: DQN and IMPALA.

Below you can see the comparison for two different reward models (scoring and checkpoint).

We can see how increasing the level of difficulty requires the models to use a larger number of steps.

The Football AcademyThe Football Engine allow us to simulate and entire football game while the Football Benchmark allow to evaluate different reinforcement learning models against well-established challenges.

The final step might be to learn how to train reinforcement learning agents for the Football Benchmark.

That’s the role of the Football Academy, a diverse set of scenarios of varying difficulty which its main goal is to allow researchers to get started on new ideas quickly, and iterate on them.

The Football Academy includes diverse settings where agents have to learn how to score against an empty goal, how to run towards a keeper, how to quickly pass in between players to defeat the defense line, or how to execute a fast counterattack.

For instance, below we can see the results of evaluating the IMPALA algorithm across the different scenarios of the Football Academy.

To see how the Football Engine, Benchmark and Academy come together, you can watch the following video released by the Google Brain team.

Google Research’s Football Environment is one of the most ambitious reinforcement learning projects up to date.

The stack will allow AI researchers to evaluate ideas against one of the most challenging reinforcement learning environments in existence and one that is connected with strong human emotions.


Reposted with permission.

Bio: Jesus Rodriguez is a technology expert, executive investor, and startup advisor.

 A software scientist by background, Jesus is an internationally recognized speaker and author with contributions that include hundreds of articles and presentations at industry conferences.

Related: var disqus_shortname = kdnuggets; (function() { var dsq = document.

createElement(script); dsq.

type = text/javascript; dsq.

async = true; dsq.

src = https://kdnuggets.



js; (document.

getElementsByTagName(head)[0] || document.


appendChild(dsq); })();.

. More details

Leave a Reply