I am starting with the generic TensorFlow example.

To classify my data I need to use multiple labels (ideally multiple softmax classifiers) on the final layer, because my data carries multiple independent labels (sum of probabilities is not 1).

Specifically in the retrain.py these lines in add_final_training_ops() add the final tensor

final_tensor = tf.nn.softmax(logits, name=final_tensor_name)

and here

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(
      logits, ground_truth_input)

Is there already a generic classifier in TensorFlow? If not, how to achieve multilevel classification?

add_final_training_ops() from tensorflow/examples/image_retraining/retrain.py:

def add_final_training_ops(class_count, final_tensor_name, bottleneck_tensor):

  with tf.name_scope('input'):
    bottleneck_input = tf.placeholder_with_default(
        bottleneck_tensor, shape=[None, BOTTLENECK_TENSOR_SIZE],

    ground_truth_input = tf.placeholder(tf.float32,
                                        [None, class_count],

  layer_name = 'final_training_ops'
  with tf.name_scope(layer_name):
    with tf.name_scope('weights'):
      layer_weights = tf.Variable(tf.truncated_normal([BOTTLENECK_TENSOR_SIZE, class_count], stddev=0.001), name='final_weights')
    with tf.name_scope('biases'):
      layer_biases = tf.Variable(tf.zeros([class_count]), name='final_biases')
    with tf.name_scope('Wx_plus_b'):
      logits = tf.matmul(bottleneck_input, layer_weights) + layer_biases
      tf.summary.histogram('pre_activations', logits)

  final_tensor = tf.nn.softmax(logits, name=final_tensor_name)
  tf.summary.histogram('activations', final_tensor)

  with tf.name_scope('cross_entropy'):
    cross_entropy = tf.nn.softmax_cross_entropy_with_logits(
      logits, ground_truth_input)
    with tf.name_scope('total'):
      cross_entropy_mean = tf.reduce_mean(cross_entropy)
  tf.summary.scalar('cross_entropy', cross_entropy_mean)

  with tf.name_scope('train'):
    train_step = tf.train.GradientDescentOptimizer(FLAGS.learning_rate).minimize(

  return (train_step, cross_entropy_mean, bottleneck_input, ground_truth_input,

Even after adding the sigmoid classifier and retraining, Tensorboard still shows softmax:

Tensorboard with softmax

  • $\begingroup$ Are all the labels in your data independent classes, or is there a mix where some groups are mutually exclusive? $\endgroup$ – Neil Slater Dec 12 '16 at 8:23
  • 1
    $\begingroup$ If all classes are exclusive, then you can just use softmax. Saying "all classes are exclusive" is same as saying "class probabilities sum to 1". Saying "all classes are independent" is the opposite. $\endgroup$ – Neil Slater Dec 12 '16 at 8:52
  • $\begingroup$ Okay, so the problem is that I have independent classes that are used based on the outcome of the softmax classifier. In my data there is a mix of 3 mutually exclusive groups of classes. Based on the result of this softmax classification, certain classes will be used from a poll of independent classes. For simplification of this question, you may assume all classes are independent. $\endgroup$ – Peter Gerhat Dec 12 '16 at 9:35
  • $\begingroup$ If all classes are independent, then you can use sigmoid activation in output layer to represent the class probabilities. So for a simplified approach you can just swap softmax for sigmoid. There is even a sigmoid_cross_entropy_with_logits function so the example can be adapted very easily. Dealing with the hierarchical class memberships would be more complex. Your question looks a bit messy BTW, it is not clear why you have pasted all that code from the examples? $\endgroup$ – Neil Slater Dec 12 '16 at 9:44
  • $\begingroup$ I removed some of the code, because I am wondering where to switch from softmax to sigmoid. Simply using final_tensor = tf.nn.sigmoid(logits, name=final_tensor_name) and cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits( logits, ground_truth_input) does not change the classifier. What am I doing wrong? $\endgroup$ – Peter Gerhat Dec 12 '16 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.