2
$\begingroup$

Given the significant advancements in reinforcement learning, I wanted to know whether it is possible to recast problems such as action recogniton, object tracking, or image classification into reinforcement learning problems.

$\endgroup$

1 Answer 1

1
$\begingroup$

Given the significant advancements in reinforcement learning

Worth noting that many of the recent advancements are due to improvements in neural networks used as function approximators, and understanding how to integrate them with reinforcement learning (RL) to help solve RL challenges involving vision or other complex non-linear mapping from state to best action.

So at least some of current improvements in RL were due to researchers asking the opposite question "Given the significant advancements in neural networks . . ."

I wanted to know whether it is possible to recast problems such as action recogniton, object tracking, or image classification into reinforcement learning problems.

Generally, at the top level, the answer is Yes, but it offers no benefit, and could perform a lot worse.

That is because in typical supervised learning scenarios there is nothing to match the concept of actions that modify state. In the example classifiers you can have the equivalent of a state (the input to be classified), an action (the choice of category) and the reward (whether of not the choice matches the label). But taking an action does not lead to another state, rewards are not sparse or cumulative across multiple actions within a specific environment or "episode". There are no time steps.

RL algorithms are generic MDP solvers - they can learn about relationships between state, action and likely next state, and optimise long-term goals over may time steps when given a current state. That also makes them less efficient learners when those relationships are not valid or important in the problem you want to solve. If you trained e.g. Q-learning on a typical image classification dataset, and added time steps, it would spend a lot of time/resources establishing that its choice of action had no influence on which images it was subsequently presented with, or how easy it was to get a reward from later images as opposed to earlier ones depending on how the state varied. If you really did allow the choice of action to determine the next image, then you would be training the RL to do something else other than classification.

You could frame the classifiers as contextual bandits, which is perhaps closer match. However, that still throws away knowledge that you have about the classification problem, replacing it with a generic reward system. For instance, a contextual bandit solver would deliberately guess wrong classes to check whether sometimes there was a small chance of a high reward for doing that.

If you were very careful about how you represented actions and rewards, and set other hyper-parameters, then you might be able to re-create a similar gradient setup to normal supervised learning, and only lose a little bit of efficiency through using RL or contextual bandit framing of your problem. However, you would still have added some non-necessary complexity.

If you search, you may find some ways to combine RL with supervised learning, for instance in this paper, the authors propose using RL to refine a generative RNN. However, these currently seem niche and are not aimed to improve upon or replace supervised learning.

Finally, in theory you could allow RL to control a video camera pan/zoom as part of activity recognition (or any other video or multi-image classification task). That would be a full RL problem, because the agent's actions really would be influencing the later states and hopefully improving the accuracy of recognition. For learning efficiency, you would likely want to combine this initially with a network that had already been trained to recognise actions on a supervised data set. You would need to experiment with how much the recognition part was trained compared to the RL part (as it will start to collect data outside of your normal datasets). And of course setup and training of the combined system could be a major project. You may be able to simulate it in a game engine perhaps in the early stages.

$\endgroup$
2
  • $\begingroup$ Hi Niel, you have answered some of my questions on reinforcement learning in the past. I wanted to ask you if you had any ideas about this question:ai.stackexchange.com/questions/8971/… $\endgroup$
    – piccolo
    Nov 14, 2018 at 13:17
  • 1
    $\begingroup$ @ImranRashid You can be fairly sure that if you ask a RL question on Data Science, Cross Validated or AI, that I will see it within a couple of days. I have already seen yours and upvoted because I think it is interesting. However, I don't know HER, so cannot answer it for now. $\endgroup$ Nov 14, 2018 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.