1
$\begingroup$

I'm trying to solve the CartPole-v1 problem from OpenAI by using backprop on a one-layer neural network - while updating the model at every time step using State action values (Q(s,a)). I'm unable to get the average reward to go up beyond about 42 steps per episode. Could anyone help? I've tried looking for a similar solution online but none of them seem to fit the approach that I'm following. Almost all of the solutions involve learning at the end of each episode (by storing SARS' data). Is my approach even correct - as in, is it even possible for the agent to learn the optimal solution if I'm updating the Q-values every time-step, instead of batch updates every episode? Seems like theoretically it should be possible.

Details: After playing around and experimenting with activation functions, stochastic policies and finally settling on a deterministic policy with linear activation function and the parameters mentioned below - i'm able to get my agent to consistently converge (in about 100-300 steps) to an average reward of about 42 steps. But it doesn't go beyond 45. Adjusting the parameters (epsilon, discount_rate, and learning rate) in the program below does not have a huge impact on this.

Here's some snippets from my code below. Basically it consists of 4 features -> 6 hidden layer nodes -> 2 outputs (one for each action).

First, the hyperparameters:

epsilon = 0.5
lr = 0.05
discount_rate=0.9

# number of features in environment observations
num_inputs = 4 
hidden_layer_nodes = 6
num_outputs = 2

The q function:

def calculateNNOutput(observation, m1, m2):
    scaled_observation = scaleFeatures(observation)
    hidden_layer = np.dot(scaled_observation, m1)           # 1x4 X 4x6 -> 1x6 
    outputs = np.dot(hidden_layer, m2) # 1x6 X 6x2  
    return np.asmatrix(outputs) # 1x2

Action selection (policy):

def selectAction(observation):
    #explore
    global epsilon
    if random.uniform(0,1) < epsilon:
        return random.randint(0,1)
    #exploit
    outputs = calculateNNOutputs(observation)
    print(outputs)
    if (outputs[0,0] > outputs[0,1]):
        return 0
    else: 
        return 1

Backprop:

def backProp(prev_obs, m1, m2, experimental_values):
    global lr
    scaled_observation = np.asmatrix(scaleFeatures(prev_obs))
    hidden_layer = np.asmatrix(np.dot(scaled_observation, m1))      # 
    outputs = np.asmatrix(np.dot(hidden_layer, m2)) # 1x6 X 6x2
    delta_out = np.asmatrix((outputs-experimental_values)) # 1x2
    delta_2=np.transpose(np.dot(m2,np.transpose(delta_out))) # 6x2 X 2x1 = 6x1_T = 1x6
    GRADIENT_2 = (np.transpose(hidden_layer))*delta_out # 6x1 X 1x2 = 6x2 - same as w2
    GRADIENT_1 = np.multiply(np.transpose(scaled_observation), delta_2) # 4 x 6 - same as w1

    m1 = m1 - lr*GRADIENT_1
    m2 = m2 - lr*GRADIENT_2
    return m1, m2

Q-learning:

def updateWeights(prev_obs, action, obs, reward, done):
    global weights_1, weights_2
    calculated_value = calculateNNOutputs(prev_obs)
    if done: 
        experimental_value = -1
    else:
        actionValues = calculateNNOutputs(obs) # 1x2
        experimental_value = reward +  discount_rate*(np.amax(actionValues, axis = 1)[0,0])
    if action==0:
        weights_1, weights_2 = backProp(prev_obs, weights_1, weights_2, np.array([[experimental_value, calculated_value[0,1]]]))
    else:
        weights_1, weights_2 = backProp(prev_obs, weights_1, weights_2, np.array([[calculated_value[0,0],experimental_value]]))

Do I need to make any changes in approach/implementation/parameter tuning?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.