I'm trying to solve the CartPole-v1 problem from OpenAI by using backprop on a one-layer neural network - while updating the model at every time step using State action values (Q(s,a)). I'm unable to get the average reward to go up beyond about 42 steps per episode. Could anyone help? I've tried looking for a similar solution online but none of them seem to fit the approach that I'm following. Almost all of the solutions involve learning at the end of each episode (by storing SARS' data). Is my approach even correct - as in, is it even possible for the agent to learn the optimal solution if I'm updating the Q-values every time-step, instead of batch updates every episode? Seems like theoretically it should be possible.
Details: After playing around and experimenting with activation functions, stochastic policies and finally settling on a deterministic policy with linear activation function and the parameters mentioned below - i'm able to get my agent to consistently converge (in about 100-300 steps) to an average reward of about 42 steps. But it doesn't go beyond 45. Adjusting the parameters (epsilon, discount_rate, and learning rate) in the program below does not have a huge impact on this.
Here's some snippets from my code below. Basically it consists of 4 features -> 6 hidden layer nodes -> 2 outputs (one for each action).
First, the hyperparameters:
epsilon = 0.5 lr = 0.05 discount_rate=0.9 # number of features in environment observations num_inputs = 4 hidden_layer_nodes = 6 num_outputs = 2
The q function:
def calculateNNOutput(observation, m1, m2): scaled_observation = scaleFeatures(observation) hidden_layer = np.dot(scaled_observation, m1) # 1x4 X 4x6 -> 1x6 outputs = np.dot(hidden_layer, m2) # 1x6 X 6x2 return np.asmatrix(outputs) # 1x2
Action selection (policy):
def selectAction(observation): #explore global epsilon if random.uniform(0,1) < epsilon: return random.randint(0,1) #exploit outputs = calculateNNOutputs(observation) print(outputs) if (outputs[0,0] > outputs[0,1]): return 0 else: return 1
def backProp(prev_obs, m1, m2, experimental_values): global lr scaled_observation = np.asmatrix(scaleFeatures(prev_obs)) hidden_layer = np.asmatrix(np.dot(scaled_observation, m1)) # outputs = np.asmatrix(np.dot(hidden_layer, m2)) # 1x6 X 6x2 delta_out = np.asmatrix((outputs-experimental_values)) # 1x2 delta_2=np.transpose(np.dot(m2,np.transpose(delta_out))) # 6x2 X 2x1 = 6x1_T = 1x6 GRADIENT_2 = (np.transpose(hidden_layer))*delta_out # 6x1 X 1x2 = 6x2 - same as w2 GRADIENT_1 = np.multiply(np.transpose(scaled_observation), delta_2) # 4 x 6 - same as w1 m1 = m1 - lr*GRADIENT_1 m2 = m2 - lr*GRADIENT_2 return m1, m2
def updateWeights(prev_obs, action, obs, reward, done): global weights_1, weights_2 calculated_value = calculateNNOutputs(prev_obs) if done: experimental_value = -1 else: actionValues = calculateNNOutputs(obs) # 1x2 experimental_value = reward + discount_rate*(np.amax(actionValues, axis = 1)[0,0]) if action==0: weights_1, weights_2 = backProp(prev_obs, weights_1, weights_2, np.array([[experimental_value, calculated_value[0,1]]])) else: weights_1, weights_2 = backProp(prev_obs, weights_1, weights_2, np.array([[calculated_value[0,0],experimental_value]]))
Do I need to make any changes in approach/implementation/parameter tuning?