Adding a bias makes Q-learning algorithm ineffective

Question

I've been working through the Q-Network learning example in this Arthur Juliani's blog. It's based on the pretty trivia Open Gym Frozen Lake example. It's base implementation get's about 47% success rate over 3000 iterations. I decided to add a bias to the implementation, and found that it severely harmed the results to no better than random.

That is, I added the bias term below:

inputs1 = tf.placeholder(shape=[1,16],dtype=tf.float32)
bias = tf.Variable(tf.zeros(shape=[1,4]))
W = tf.Variable(tf.random_uniform([16,4],0,0.01))
Qout = tf.matmul(inputs1,W) + bias
predict = tf.argmax(Qout,1)

The rest of the code is identical to the original solution. Any ideas why this would so negatively affect performance?

Update It looks like someone else ran into this issue, and the answer given was that

Having a bias term with the one-hot encoding prevents each state’s Q values from being independent

Any ideas why this is the case? The bias is added after the multiplication, so it's in the dimension of the actions, not the inputs. I don't see why this would make learning fail.

Answer 1

Having a bias term with the one-hot encoding prevents each state’s Q values from being independent

Any ideas why this is the case?

The usual point of using linear regression or other function approximation in Q-learning is to generalise, and thus prevent the Q values from being independent deliberately. So this is not a general statement about Q-learning and linear function approximation. In general yes you can add bias terms - also, in general when using linear regression or neural networks to estimate action values, then you should have bias terms.

Having said that, I would expect the bias should make little difference here, since the one hot coding of state makes it redundant - you can already express any mapping of state to Q value of each possible action using just the weights. Although you are using what looks like a linear regression model, essentially this is just the tabular form of Q learning.

Q learning with function approximation can be unstable. The combination of off-policy, bootstrap updates and function approximation is known as the "deadly triad", and you need to add tricks like experience replay to keep it stable. Although in theory the linear regression should learn a correct bias, probably adding bias puts the algorithm into that unstable zone.