Reinforcement learning

[jarnmo]

I've looked into the available documentation and examples, but haven't been able to figure out if it is possible to use the ML.NET in its current state for (non-deep) reinforcement learning. If it is possible, I'd be thankful for any hints on how to implement a simple case. In case reinforcement learning is not possible atm, what exactly is missing and are there any plans on implementing the missing pieces?

Thanks!

[GalOshri]

Reinforcement learning is not yet available in ML.NET. One critical component that is currently missing is "exploration" (trying out different actions to see how the reward changes) and how this affects data collection and model training. This is something that would be interesting to explore in the future.

What is the scenario you have in mind?

[jarnmo]

I see. My scenario is basically to use it as a part of building a game AI.

[GalOshri]

Got it. A few more questions: what is the complexity of the game? How many actions does the AI have (the number of choices available to it at each step)? Does the list of actions change throughout the course of the game or is it static?

[jarnmo]

Well, what I'm actually trying to do, is to build a framework which would be used to write game rules (gameplay code) and would make it as easy as possible to add AI actors as well. In the end anything but a trivial AI would probably have to be build of several components, some ML and some utilizing more traditional features of the framework. The end goal is to allow building games of any practical complexity, but no 'academic restrictions' apply here. RL can only be a piece in the AI puzzle and for example 'goal hacking' is certainly an option if it makes the AI behave in a desirable way :).

While a game AI can be extremely complex, I'm hoping to find ways to split the problem domains into smaller pieces that could then be handled by individual 'RL pipelines'. Maybe even just learn to estimate the value of taking each action in a given situation individually. I think the Microsoft's Mrs PacMan AI used this kind of an approach.

It's certainly common that the list of possible actions an actor can take changes throughout the course of a game, but if the value of each action (for each target) would be evaluated by it's own NN I guess the changes would be a matter of updating the list of evaluated NNs (and whatever logic coordinates between them).

This is all very much just an idea atm, but I guess even a very simple RL implementation would allow meaningful experimentation. I might look into CNTK and their C# bindings now, but as I have no practical experience from ML I wouldn't mind something a bit easier to approach..

[axmand]

To Implement RL such as DQN, I need to train neural network after an observation.
however, i don't know how to split train data to train the neural network by 'TextLoader'.
Is there any way to gradually input training data?
//
well, i found this in:

machinelearning/test/Microsoft.ML.Tests/CollectionDataSourceTests.cs

Line 29 in f6c6f5b

Assert.NotNull(CollectionDataSource.Create(new List<Input>() { new Input { Number1 = 1, String1 = "1" } }));

Any news about some RL support?

[codemzs]

CC: @wschin

[chris3713]

Would be nice too see this implemented some time.

[codemzs]

We don't have RL support in the short-term but it is something we are considering for long-term.