In deep learning, we can assess model's performance with loss function value and improve model's performance with K-fold cross-validation and so on. But how can we design and tune neural network used in deep reinforcement learning? We can assess reinforcement learning algorithm's performance itself with rewards and so on, but how can we be sure that neural network used in reinforcement learning algorithm is good or bad ?
1
-
$\begingroup$ When you use a neural (agent / policy) network it is trained using the reward signal and there is no meaningful way to assess it's performance independently. However you may pair a neural network with a classical RL agent. For example the neural network can look at images of football players and then output a vector predicting the trajectory of each opponent at the next few time steps. The agent can then treat this vector as an observation and learn to maximize its reward (dodging/passing/scoring) based on the observation. The network can be trained and evaluated independently of the agent. $\endgroup$– farhanhubbleCommented Jul 28, 2023 at 5:53
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
We can assess reinforcement learning algorithm's performance itself with rewards and so on, but how can we be sure that neural network used in reinforcement learning algorithm is good or bad ?
The "goodness" of the neural network is exactly what's being communicated to us through the reward signal from the environment. After all, the neural net (at least indirectly) determines what actions the agent takes. Any suboptimal behavior would result in lower reward. This information is backpropagated to the weights of the neural net by the RL algorithm being used.
For example, if we're using a value network and we're trying to minimize error between experienced and predicted return, then we can perform stochastic gradient descent on parameters $w$
$$\begin{align}w_{t+1} &\doteq w_t - \frac{1}{2}\alpha\nabla_{w_t}\left[G_t-\hat{v}(S_t,w_t)\right]^2\\ &= w_t + \alpha\left[G_t-\hat{v}(S_t,w_t)\right]\nabla_{w_t}\hat{v}(S_t,w_t) \end{align}$$
(Once the neural net is fairly good at predicting the outcome of its actions, it can begin to optimize its behavior through Generalized Policy Iteration.)
answered Mar 11, 2019 at 10:42
$\begingroup$
$\endgroup$
Neural networks used in Reinforcement learning (RL) models can be evaluated by amount of reward over time. A neural network that accumulates more reward is better than another neural network that accumulates less reward in the same amount of time.
The process of experimenting and improving neural networks for RL can often take longer because the learning process for RL is typically longer.
answered Feb 18, 2022 at 13:01
$\begingroup$
$\endgroup$
1
The process of reinforcement learning already implies that you have a base model to work from, that's what you're reinforcing. So, presumably, that underlying model is already good, otherwise you wouldn't be using it, right?
The whole point of reinforcement learning is to introduce your (functioning) model to new information and/or changing conditions. Reinforcement learning isn't going to take a bad model and turn it into a good one. If your base model is not performing to your satisfaction, then you should go back to a model design/selection phase; reinforcement learning is not going to help you in that scenario.
answered Feb 8, 2019 at 15:55
-
2$\begingroup$ I didn't wish to downvote the answer as it seems very misleading, so I prefer to ask for clarifications. Which underlying model you are referring to? In RL you did not reinforce any kind of model. It is called RL because you reinforce rewarding behaviors and the ones that are being reinforced (by rewards) are the ones that are going to be learned. You either have a RL model that you train from scratch or else you are talking about something else. $\endgroup$ Commented Feb 9, 2019 at 19:16