2
$\begingroup$

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )? I want to replace the DNN section of Qlearning with a RF or SVR but the problem is that there is no clear training data that I can put in my code at tensorflow or keras! How we can do this?

$\endgroup$
0

1 Answer 1

1
$\begingroup$

You would need to train the RF-QN or SVR-QN on a very large batch/sample generated in the same way as a mini-batch in the DQN version. The input data would be the states and actions visited whilst simulating or running the environment for the batch, and the label would be the TD Target $r + \gamma \text{max}_{a'}\hat{q}(s', a')$ using an old copy of the RF or SVR model to calculate $\hat{q}$ with a loss function of MSE.

The reason you would need a very large batch (and why you generally don't see this done, because it will be far slower than NN) is because neither RF nor SVR in their basic form can be made to run online. However, if you manage to find online versions of the algorithm (I know there are some for RF), then you can use them almost identically to a DNN.

Wherever you see $\hat{q}$ in the pseudo-code, you know you need to run the model forward to estimate Q values. Typically you need to do this to make action selections (by finding best Q value for possible actions) and to generate TD Targets for further training.

Here's how you would need to change a DQN implementation in general terms:

  • You need two copies of your SVR or RF model: a "target model" and a "learning model". You would start with a basic target model that can predict either random values or fixed values for $Q(s,a)$ - those will clearly be wrong, but it should not matter. Each model will train using output from the previous one, plus the real data about transitions and reward. On each training session, the model becomes more accurate, and allows for better selection of actions.

  • Wherever you see a call to predict or train the DNN, replace with same call to predict or train the SVR or RF. When you see the current network being cloned to the "target network" do the same with the SVR or RF model. Pay attention to when the code uses the learning network or the target network, and use the appropriate SVR or RF model.

  • If you are using the default full batch training of RF or SVR, then you will need to generate a large dataset to train at once, and you should clone the resulting newly trained SVR or RF model direct to the target model immediately. Also, as the learning model in SVR or RF does not work online, you will need to use the target network to select actions in your case - that should be fine, although it may slow down learning further. Depending on your RL problem, without online versions of the algorithms, your dataset could easily need to be 100,000 or more records per training and cloning step.

$\endgroup$
4
  • $\begingroup$ minibatch = random.sample(self.memory, batch_size) for state, action, reward, next_state, done in minibatch: target = reward if not done: target = (reward + self.gamma * np.amax(self.model.predict(next_state)[0])) target_f = self.model.predict(state) target_f[0][action] = target self.model.fit(state, target_f, epochs=1, verbose=0) if self.epsilon > self.epsilon_min: self.epsilon *= self.epsilon_decay $\endgroup$ Commented Aug 31, 2018 at 20:50
  • $\begingroup$ Thanks Neil! would you please say what I should do if 1- with experience replay 2-without experience replay! With experience replay we train but what what we could say when we do not use it! For instance in above code of experience replay, How I can said the Xtrain Ytrain? $\endgroup$ Commented Aug 31, 2018 at 20:51
  • $\begingroup$ @user10296606: From above code, Xtrain is an array of state, and Ytrain is the matching array of target_f (and inside the for loop shows how just one of the example rows is built). The model from your code example has been designed to predict an array of Q values covering all possible actions from a single state e.g. $[\hat{q}(s,a_0),\hat{q}(s,a_1), \hat{q}(s,a_2) ... \hat{q}(s,a_N)]$ - that is one of two possible choices for how to structure the predictive model for Q. I expect SVR and RF can do that too, but if not, you need to change more (or find a closer example). $\endgroup$ Commented Sep 1, 2018 at 8:12
  • $\begingroup$ @user10296606: I cannot really analyse and respond to code in comments though, so if you are still stuck this needs another question with code. Please have a try yourself first though, and only ask about places where you are stuck. I won't have time to do the whole conversion to RF for you. $\endgroup$ Commented Sep 1, 2018 at 8:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.