1
$\begingroup$

When tuning the Deep Q net parameters we use the immediate rewards . Specially in action replay and regressing .

But in the run time we don't care about the rewards at all. Because our neural network will take states as the input . I am explaining this relevant to the deep mind Atari game play paper . Any one thing it is wastage of input ? Is there a mechanism we can use something like Score even in the run time .

p.s - I understand how they update the Neural Net parameters with regression . Sometimes with TD(λ) .

$\endgroup$
1
$\begingroup$

My friend - Thushan Gave me this answer . So I will post it . I think it will describe it . Sometimes in the run time it's not desirable to work with immediate rewards . It can mess up the agent. I will quote it here .

Your argument makes sense, but not always. And this is a design decision you make in your problem formulation. However to highlight why this won't work in some situations, I'll give an example.

Say you're playing a shooting game with a RL agent. And at some specific location you get killed by getting shot from an enemy. However if you include the reward with the state, this might suggest to the algorithm, that this location is always bad and will try to avoid it (which is wrong). We should't be avoiding locations, but enemies.

$\endgroup$
0
$\begingroup$

I am not very sure what do you mean as input. In Supervised Learning the learning signal comes from the difference between true response and model's response ("teacher's supervision"). In Reinforcement Learning the learning signal comes from the reward which might come delayed, sometimes not at all in that particular trial (but in another yes) etc.

Deep-Q learning which basically is Q-learning with function approximation is a Model-free RL. It means that at the end you want your system to learn a mapping between stimulus and response. Think of it as a reflex elicited by a stimulus. The reward cannot be the input to your system as it is your learning signal.

If you are referring to the experience replay, as I mentioned to you, the reward sometimes doesn't come "on time". So we need to decorrelate states,actions and sequences and that's why we don't update the network at every single time step. Instead we prefer to build a buffer with experience and sample from that. So as you stated, you want to learn to avoid enemies not locations and for this if you sample experience from the buffer the network's training will be more "intuitive".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.