1
$\begingroup$

On lines 142 and 143 of: https://github.com/nyck33/openai_spinup_my_implements/blob/master/continuous/mountaincar/my_ddpg_ac.py

I have:

self.get_action_gradients = K.function(inputs=[self.model.input[0], self.model.input[1], \
            K.learning_phase()], outputs=[action_gradients])

Which tells me:
line 143, in build_model K.learning_phase()], outputs=[action_gradients])

TypeError: Can not convert a list into a Tensor or Operation.

action_gradients are calculated on line 140 via:

action_gradients = K.gradients(Q_value, actions)

so I did not think that is a problem but when I take the brackets off of the output argument for K.function like so:

self.get_action_gradients = K.function(inputs=[*self.model.input, \
            K.learning_phase()], outputs=action_gradients)

Now I get a slightly different error mentioning Nonetype rather than list:

rning-copied/_1my_imps/continuous/mountaincar/my_ddpg_ac.py", line 143, in build_model

TypeError: Can not convert a NoneType into a Tensor or Operation.

Printing out the Q_values and actions show they are tensors:

Q_values Tensor("q_values/BiasAdd:0", shape=(?, 1), dtype=float32) actions Tensor("actions:0", shape=(?, 1), dtype=float32)

But printing out the action_gradients and type(action_gradients) just confuses me more:

action_gradients [None]
action_gradients type <class 'list'>

calling: K.gradients() on two tensors should work shouldn't it?

This DDPG code is originally from here: https://github.com/nyck33/autonomous_quadcopter

and I am trying to adapt it for MountainCarContinuous-v0

$\endgroup$

1 Answer 1

0
$\begingroup$

I am very sorry but this was a careless error. In the critic network I had different pathways for state and action, joining them right before the Q-value output layer but...

I actually had state as the input for both pathways rather than action for the action pathway.

So of course, if the action is not present in a network, there is no way to use the chain rule to get the dQ(s,a)/da gradient of output w.r.t. actions.

Here is the code for clarification:

def build_model(self):
    #lrelu = LeakyReLU(alpha=0.1)
    #Define input layers
    states = layers.Input(shape=(self.state_size,), name="states")
    actions = layers.Input(shape=(self.action_size,), name="actions")

    #Add hidden layers for state pathway
    net_states = layers.Dense(units=32, use_bias=False)(states)
    net_states = layers.BatchNormalization()(net_states)
    net_states = layers.LeakyReLU(alpha=0.1)(net_states)

    net_states = layers.Dense(units=64)(net_states)
    net_states = layers.BatchNormalization()(net_states)
    net_states = layers.LeakyReLU(alpha=0.1)(net_states)

    #hidden layers for action
    net_actions = layers.Dense(units=32)(actions) #had (states) here instead
    net_actions = layers.BatchNormalization()(net_actions)
    net_actions = layers.LeakyReLU(alpha=0.1)(net_actions)

    net_actions = layers.Dense(units=64)(net_actions)
    net_actions = layers.BatchNormalization()(net_actions)
    net_actions = layers.LeakyReLU(alpha=0.1)(net_actions)

and the output to console shows that it is no longer a None type:

action_gradients [<tf.Tensor 'gradients_3/dense_13/MatMul_grad/MatMul:0' shape=(?, 1) dtype=float32>]

Edit: The code works for MtnCarContinuous-v0 environment of openai gym with a modified reward function (incremental based on car position because the final reward is so sparse), over 90 seems to be achieved in 11 episodes. I rendered so actually saw the car make it to the top on the attempts where score is over 100 below. Here are the results:

episode: 0 score: -41.59034754863541 mean: -41.59 std: 0.0
episode: 1 score: -75.87242108451743 mean: -58.73 std: 17.14
episode: 2 score: 32.01844640263133 mean: -28.48 std: 45.01
episode: 3 score: 132.904319261567 mean: 11.86 std: 80.02
episode: 4 score: 125.81198290946529 mean: 34.65 std: 84.85
episode: 5 score: 84.26163480413017 mean: 42.92 std: 79.63
episode: 6 score: 126.89684490110164 mean: 54.92 std: 79.37
episode: 7 score: 139.18190524840517 mean: 65.45 std: 79.3
episode: 8 score: 100.24481691450521 mean: 69.32 std: 75.56
episode: 9 score: 165.80286734425076 mean: 78.97 std: 77.31
episode: 10 score: 109.29507292352991 mean: 94.05 std: 66.24
episode: 11 score: 209.07900825070152 mean: 122.55 std: 44.84

Source code here: https://github.com/nyck33/openai_my_implements/tree/master/continuous/mountaincar

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.