3
$\begingroup$

In deep learning, we can assess model's performance with loss function value and improve model's performance with K-fold cross-validation and so on. But how can we design and tune neural network used in deep reinforcement learning? We can assess reinforcement learning algorithm's performance itself with rewards and so on, but how can we be sure that neural network used in reinforcement learning algorithm is good or bad ?

$\endgroup$
0
$\begingroup$

The process of reinforcement learning already implies that you have a base model to work from, that's what you're reinforcing. So, presumably, that underlying model is already good, otherwise you wouldn't be using it, right?

The whole point of reinforcement learning is to introduce your (functioning) model to new information and/or changing conditions. Reinforcement learning isn't going to take a bad model and turn it into a good one. If your base model is not performing to your satisfaction, then you should go back to a model design/selection phase; reinforcement learning is not going to help you in that scenario.

$\endgroup$
1
  • 1
    $\begingroup$ I didn't wish to downvote the answer as it seems very misleading, so I prefer to ask for clarifications. Which underlying model you are referring to? In RL you did not reinforce any kind of model. It is called RL because you reinforce rewarding behaviors and the ones that are being reinforced (by rewards) are the ones that are going to be learned. You either have a RL model that you train from scratch or else you are talking about something else. $\endgroup$ Feb 9 '19 at 19:16
0
$\begingroup$

We can assess reinforcement learning algorithm's performance itself with rewards and so on, but how can we be sure that neural network used in reinforcement learning algorithm is good or bad ?

The "goodness" of the neural network is exactly what's being communicated to us through the reward signal from the environment. After all, the neural net (at least indirectly) determines what actions the agent takes. Any suboptimal behavior would result in lower reward. This information is backpropagated to the weights of the neural net by the RL algorithm being used.

For example, if we're using a value network and we're trying to minimize error between experienced and predicted return, then we can perform stochastic gradient descent on parameters $w$

$$\begin{align}w_{t+1} &\doteq w_t - \frac{1}{2}\alpha\nabla_{w_t}\left[G_t-\hat{v}(S_t,w_t)\right]^2\\ &= w_t + \alpha\left[G_t-\hat{v}(S_t,w_t)\right]\nabla_{w_t}\hat{v}(S_t,w_t) \end{align}$$

(Once the neural net is fairly good at predicting the outcome of its actions, it can begin to optimize its behavior through Generalized Policy Iteration.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.