I'm trying to learn a simple linear softmax model on some data. The LogisticRegression in scikit-learn seems to work fine, and now I am trying to port the code to TensorFlow, but I'm not getting the same performance, but quite a bit worse. I understand that the results will not be exactly equal (scikit learn has regularization params etc), but it's too far off.

total = pd.read_feather('testfile.feather')

labels = total['labels']
features = total[['f1', 'f2']]


classifier = linear_model.LogisticRegression(C=1e5, solver='newton-cg', multi_class='multinomial')
classifier.fit(features, labels)
pred_labels = classifier.predict(features)

print('\tAccuracy:', classifier.score(features, labels)) 
print('\tPrecision:', precision_score(labels, pred_labels, average='macro'))
print('\tRecall:', recall_score(labels, pred_labels, average='macro'))
print('\tF1:', f1_score(labels, pred_labels, average='macro'))

# now try softmax regression with tensorflow 

## By default, the OneHotEncoder class will return a more efficient sparse encoding. 
## This may not be suitable for some applications, such as use with the Keras deep learning library. 
## In this case, we disabled the sparse return type by setting the sparse=False argument.
enc = OneHotEncoder(sparse=False)
enc.fit(labels.values.reshape(len(labels), 1)) # Reshape is required as 
Encoder expect 2D data as input
labels_one_hot = enc.transform(labels.values.reshape(len(labels), 1))

# tf Graph Input
x = tf.placeholder(tf.float32, [None, 2]) # 2 input features
y = tf.placeholder(tf.float32, [None, 5]) # 5 output classes

W = tf.Variable(tf.zeros([2, 5]))
b = tf.Variable(tf.zeros([5]))

# Construct model
pred = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax

clas = tf.argmax(pred, axis=1)

# Minimize error using cross entropy
cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))
# Gradient Descent
optimizer = tf.train.GradientDescentOptimizer(0.01).minimize(cost)

# Initialize the variables (i.e. assign their default value)
init = tf.global_variables_initializer()

# Start training
with tf.Session() as sess:

   # Run the initializer

   # Training cycle
   for epoch in range(1000):
      # Run optimization op (backprop) and cost op (to get loss value)
    _, c = sess.run([optimizer, cost], feed_dict={x: features, y: labels_one_hot})

# Test model
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
class_out = clas.eval({x: features})

# Calculate accuracy
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print("\tAccuracy:", accuracy.eval({x: features, y: labels_one_hot}))
print('\tPrecision:', precision_score(labels, class_out, average='macro'))
print('\tRecall:', recall_score(labels, class_out, average='macro'))
print('\tF1:', f1_score(labels, class_out, average='macro'))

The output of this code is

(1681, 2)
    Accuracy: 0.822129684711
    Precision: 0.837883361162
    Recall: 0.784522522208
    F1: 0.806251963817

    Accuracy: 0.694825
    Precision: 0.735883666192
    Recall: 0.649145125846
    F1: 0.678045562185

I inspected the result of the one-hot-encoding, and the data, but I have no idea why the result in TensorFlow is much worse.

  • 2
    $\begingroup$ don't set the w weight to zeros. this will cause symmetric update problem. $\endgroup$ Jan 25, 2018 at 15:03
  • $\begingroup$ @Media is this also the case for softmax? $\endgroup$
    – Ben
    Jan 25, 2018 at 15:20
  • 1
    $\begingroup$ yes, because the partial derivatives will be same because the weights are all set to zeros. How did the result change? $\endgroup$ Jan 25, 2018 at 15:32
  • 2
    $\begingroup$ let it run for some epochs and again tell me :) $\endgroup$ Jan 25, 2018 at 16:15
  • 1
    $\begingroup$ moreover, I suggest you use xavier initialization. $\endgroup$ Jan 25, 2018 at 16:17

2 Answers 2


The problem turned out to be silly, I just needed more epochs, a smaller learning rate (and for efficiency I turned to AdamOptimizer, results are now equal.

(1681, 2)
    Accuracy: 0.822129684711
    Precision: 0.837883361162
    Recall: 0.784522522208
    F1: 0.806251963817

    Accuracy: 0.82213
    Precision: 0.837883361162
    Recall: 0.784522522208
    F1: 0.806251963817
  • 1
    $\begingroup$ I answered at the same time :D $\endgroup$ Jan 26, 2018 at 9:59

You are initializing your weights to zero. This will cause symmetric problem in updating weights. Use Xavier or He technique for initialization and train for some epochs.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.