3
$\begingroup$

I'm trying to learn a simple linear softmax model on some data. The LogisticRegression in scikit-learn seems to work fine, and now I am trying to port the code to TensorFlow, but I'm not getting the same performance, but quite a bit worse. I understand that the results will not be exactly equal (scikit learn has regularization params etc), but it's too far off.

total = pd.read_feather('testfile.feather')

labels = total['labels']
features = total[['f1', 'f2']]

print(labels.shape)
print(features.shape)

classifier = linear_model.LogisticRegression(C=1e5, solver='newton-cg', multi_class='multinomial')
classifier.fit(features, labels)
pred_labels = classifier.predict(features)

print("SCI-KITLEARN RESULTS: ")
print('\tAccuracy:', classifier.score(features, labels)) 
print('\tPrecision:', precision_score(labels, pred_labels, average='macro'))
print('\tRecall:', recall_score(labels, pred_labels, average='macro'))
print('\tF1:', f1_score(labels, pred_labels, average='macro'))

# now try softmax regression with tensorflow 
print("\n\nTENSORFLOW RESULTS: ")

## By default, the OneHotEncoder class will return a more efficient sparse encoding. 
## This may not be suitable for some applications, such as use with the Keras deep learning library. 
## In this case, we disabled the sparse return type by setting the sparse=False argument.
enc = OneHotEncoder(sparse=False)
enc.fit(labels.values.reshape(len(labels), 1)) # Reshape is required as 
Encoder expect 2D data as input
labels_one_hot = enc.transform(labels.values.reshape(len(labels), 1))

# tf Graph Input
x = tf.placeholder(tf.float32, [None, 2]) # 2 input features
y = tf.placeholder(tf.float32, [None, 5]) # 5 output classes

W = tf.Variable(tf.zeros([2, 5]))
b = tf.Variable(tf.zeros([5]))

# Construct model
pred = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax

clas = tf.argmax(pred, axis=1)

# Minimize error using cross entropy
cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))
# Gradient Descent
optimizer = tf.train.GradientDescentOptimizer(0.01).minimize(cost)

# Initialize the variables (i.e. assign their default value)
init = tf.global_variables_initializer()

# Start training
with tf.Session() as sess:

   # Run the initializer
   sess.run(init)

   # Training cycle
   for epoch in range(1000):
      # Run optimization op (backprop) and cost op (to get loss value)
    _, c = sess.run([optimizer, cost], feed_dict={x: features, y: labels_one_hot})

# Test model
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
class_out = clas.eval({x: features})

# Calculate accuracy
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print("\tAccuracy:", accuracy.eval({x: features, y: labels_one_hot}))
print('\tPrecision:', precision_score(labels, class_out, average='macro'))
print('\tRecall:', recall_score(labels, class_out, average='macro'))
print('\tF1:', f1_score(labels, class_out, average='macro'))

The output of this code is

(1681,)
(1681, 2)
SCI-KITLEARN RESULTS:
    Accuracy: 0.822129684711
    Precision: 0.837883361162
    Recall: 0.784522522208
    F1: 0.806251963817


TENSORFLOW RESULTS:
    Accuracy: 0.694825
    Precision: 0.735883666192
    Recall: 0.649145125846
    F1: 0.678045562185

I inspected the result of the one-hot-encoding, and the data, but I have no idea why the result in TensorFlow is much worse.

$\endgroup$
  • 2
    $\begingroup$ don't set the w weight to zeros. this will cause symmetric update problem. $\endgroup$ – Media Jan 25 '18 at 15:03
  • $\begingroup$ @Media is this also the case for softmax? $\endgroup$ – Ben Jan 25 '18 at 15:20
  • 1
    $\begingroup$ yes, because the partial derivatives will be same because the weights are all set to zeros. How did the result change? $\endgroup$ – Media Jan 25 '18 at 15:32
  • 2
    $\begingroup$ let it run for some epochs and again tell me :) $\endgroup$ – Media Jan 25 '18 at 16:15
  • 1
    $\begingroup$ moreover, I suggest you use xavier initialization. $\endgroup$ – Media Jan 25 '18 at 16:17
2
$\begingroup$

The problem turned out to be silly, I just needed more epochs, a smaller learning rate (and for efficiency I turned to AdamOptimizer, results are now equal.

(1681,)
(1681, 2)
SCI-KITLEARN RESULTS:
    Accuracy: 0.822129684711
    Precision: 0.837883361162
    Recall: 0.784522522208
    F1: 0.806251963817

TENSORFLOW RESULTS:
    Accuracy: 0.82213
    Precision: 0.837883361162
    Recall: 0.784522522208
    F1: 0.806251963817
$\endgroup$
  • 1
    $\begingroup$ I answered at the same time :D $\endgroup$ – Media Jan 26 '18 at 9:59
2
$\begingroup$

You are initializing your weights to zero. This will cause symmetric problem in updating weights. Use Xavier or He technique for initialization and train for some epochs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.