3
$\begingroup$

Is it possible to implement a reinforcement learning algorithm without using a deep neural network (DNN) as used in deep reinforcement learning e.g. Deep Q-Network (DQN)?

How can I replace the DNN in the DQN algorithm with another algorithm? Should it be supervised or unsupervised, and what is this called - is it "un/supervised reinforcement learning" or "un/supervised DQN"?

In the following DQN pseudo-code, if I want to replace a DNN section with another un/supervised algorithm, is it possible? If so, how?

enter image description here

Improve this question
$\endgroup$
5
  • $\begingroup$ Thanks I edit the topic what I mean is the un/supervised reinforcement learning! $\endgroup$
    – user10296606
    Aug 31, 2018 at 13:39
  • $\begingroup$ That makes sense. So you are asking if it is possible to replace the "deep" part of DQN with some other model? I will try to tidy that up without changing the meaning? $\endgroup$
    – Neil Slater
    Aug 31, 2018 at 14:43
  • $\begingroup$ Yes, I want to replace DNN section of DQN with a supervised or unsupervised learning approach? Is it possible? How? $\endgroup$
    – user10296606
    Aug 31, 2018 at 15:34
  • $\begingroup$ Yes, I want to replace DNN section of DQN with a supervised or unsupervised learning approach? Is it possible?If yes, How? Also the method is not any more DQN it will be supervised Q-Net or a unsupervised Q-Net! What you put in the title guides to a misunderstanding $\endgroup$
    – user10296606
    Aug 31, 2018 at 15:41
  • $\begingroup$ Please feel free to edit the title back to avoid misunderstanding. It was my best guess. But you want to avoid using the phrase "un/supervised reinforcement learning" because it does not mean anything (I've left it in the body of your question so that someone can answer and tell you that) $\endgroup$
    – Neil Slater
    Aug 31, 2018 at 16:06

1 Answer 1

Reset to default
1
$\begingroup$

Well, if you remove the DNN, I would not call that a Deep Q-Network anymore.. but it is definitely possible to remove that and still consider the approach as Reinforcement Learning.

Actually, the function of the deep neural network is just to approximate the Q-value function. The DNN is just a function in the form $Q_\theta(s, a)$ and based on samples, you adjust $\theta$ to minimize the error.

In practice, you could use any function you want. Obviously, some functions will work better than others. There are simpler approaches based on linear regression, least squares, kernel-based approaches, etc.

With respect to unsupervised learning.. mm. I guess you could force it somehow, but keep in mind that it would not totally make sense given that RL already provides you feedback so it makes sense that you use it.

Some interested advantage of neural networks is that they can find common elements in $\phi$ and rapidly abstract away some noise that if is not relevant with respect to predicting the Q-value of certain state/action.

I noticed that you would like to use something like random forests, etc. There is nothing to stop you from doing that. In particular, random forests are not ideal because they don't deal well with non-stationary. Instead they require bit batches of data. However, this does not mean, it is impossible.

In DQN, after you sample from the environment, you obtain feedback from the environment in the form of rewards associated with state/action pairs. RL combines this information with previous predictions using TD (there are some variations like TD($\lambda$), etc.). What TD gives you is a better estimate of $Q(s,a)$ and your predictor (in this case, your random forest) should learn that when the input is $(s,a)$ the correct answer is $Q(s,a)$.

It is likely that you've seen the same inputs before (specially for initial states) but because TD improves the reference value, you would need to drop these previous values and replace them with the most recent ones. Random forest would need to get this whole batch again.

In general, I accept that while both cases are supervised learning, adjusting DQN to work with non-parametric methods such as random forests can be inconvenient.

If I had to guess based on the comments, I think your main question circles around the distinction between parametric and non-parametric supervised learning. In that case, it is fair to say that most non-parametric methods are not easy to combine with DQN.

Improve this answer
$\endgroup$
15
  • 1
    $\begingroup$ The "true label" is the Q(s,a) value that you want to predict. Think of supervised learning for continuous functions. From the perspective of DNN, you are using supervised learning and you want to approximate the Q function as well as possible. $\endgroup$
    – Juan Leni
    Aug 31, 2018 at 15:50
  • 1
    $\begingroup$ Sorry, but I am sure about this. The training data is produced by the environment. The agent receives feedback from the environment, uses a TD update and the sampled Q value is compared with the predicted Q value. That is when the supervised learning happens. Almost any RL paper that uses a function approximator requires supervised learning. If you want a more explicit description look for any actor-critic configuration. $\endgroup$
    – Juan Leni
    Aug 31, 2018 at 16:00
  • 1
    $\begingroup$ @user10296606: You are right that you don't see RF or SVR used in Q-learning very often. It is possible in theory, but both those algorithms have a weakness when it comes to using them in RL - they do not work well against online, non-stationary datasets, whilst neural networks do. $\endgroup$
    – Neil Slater
    Aug 31, 2018 at 16:18
  • 1
    $\begingroup$ @NeilSlater well, actually it was a bit of a surprise for the community that neural networks did work. Q-Learning was previously known to converge with least squares only. $\endgroup$
    – Juan Leni
    Aug 31, 2018 at 16:21
  • 1
    $\begingroup$ @user10296606: I don't want to debate you at length in the comments. This answer is correct, so are comments by purpletentacle. If you still don't understand or disagree, then ask a new question specifically about that (e.g. "Is the neural network in DQN used to learn like a supervised model?") $\endgroup$
    – Neil Slater
    Aug 31, 2018 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.