0
$\begingroup$

I am working on a research problem where I got stuck on something which doesn't really make sense to me. To explain the issue I will give an example of a similar problem (face age detection).

Suppose that I have many face images that I precisely know how many **days** old the person in each image. (I need precision in my problem).

I train my model using CNN in three ways.

In the first training, I create a neural network that gives an output of 75 classes (say the ages are 0-74) which tells me the correct age with 30-40% accuracy. Below is the final code piece after the CNN layers:

net1 = tf.reduce_mean(net, axis=(1, 2), name='gap')

# Fully-connected classifier 
net1 = slim.fully_connected(net1, 512, activation_fn=utils.lrelu, scope='cmi_ff1')
net1 = slim.dropout(net1, prob, scope='dropout1_net1')
net1 = slim.fully_connected(net1, 256, activation_fn=utils.lrelu, scope='cmi_ff2_net1')
net1 = slim.dropout(net1, prob, scope='dropout2_net1')

y_ = slim.fully_connected(net1, n_classes, activation_fn=tf.nn.softmax, scope='cmi_ffo_net1')
loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits( logits=y_, labels=y))

n_classes=75 as the number of ages is 75.

I keep everything the same and change the output layer from classification to a regression which results in significantly lower accuracy that each person approx 37 years old. This tells me that my neural network is trying to minimize the error and doesn't learn anything. I keep everything the same except for the output layer and loss function.

y2_ = slim.fully_connected(net1, 1, activation_fn=None, scope='cmi_ffo_net2') 
loss2 = tf.reduce_mean(tf.abs(y2_ - tf.cast(y,tf.float32)))

Up to now, nothing sounds weird to me (although I am a newbie in DL). The weird thing is happening when I decide to use two heads instead of a single head. I split the network into two right after the last convolutional layer. 

net1 = tf.reduce_mean(net, axis=(1, 2), name='gap')
net2 = tf.reduce_mean(net, axis=(1, 2), name='gap')

# Fully-connected classifier 
net1 = slim.fully_connected(net1, 512, activation_fn=activation, scope='cmi_ff1')
net1 = slim.dropout(net1, prob, scope='dropout_net1')
net1 = slim.fully_connected(net1, 256, activation_fn=activation, scope='cmi_ff2_net1')
net1 = slim.dropout(net1, prob, scope='dropout_net1')

y_ = slim.fully_connected(net1, n_classes, activation_fn=tf.nn.softmax, scope='cmi_ffo_net1')
loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y_, labels=y))

# Fully-connected regressor
net2 = slim.fully_connected(net2, first_fc, activation_fn=activation, scope='cmi_ff1')
net2 = slim.dropout(net2, prob, scope='dropout_net1')
net2 = slim.fully_connected(net2, second_fc, activation_fn=activation, scope='cmi_ff2_net2')

y2_ = slim.fully_connected(net2, 1, activation_fn=None, scope='cmi_ffo_net2') 
loss2 = tf.reduce_mean(tf.abs(y2_ - tf.cast(y,tf.float32)))

Here, the accuracy of classification goes up to 65-70% and regression remains to the same (converges to the middle value). While regression seems to be useless in the entire process, somehow it gives an amazing push to the classification to improve accuracy that I can't even speculate about the reason.

I am adding a figure that shows the accuracy and loss of the two-headed network. (I was trying to optimize some hyperparameters.) figure

Explanation of the figure: 

subplot 1,1: Training accuracy for classification head
subplot 2,1: Validation accuracy for classification head
subplot 1,2: Loss for classification head
subplot 1,3: Loss for regression head
subplot 2,2: Total loss: 0.2* regression + classification

I wonder what could be the reason for this or is there something that I do wrong? Also, I know the code is not complete but I can't share the entire code. I just need some directions about what to think and where to look.

Improve this question
$\endgroup$
7
  • $\begingroup$ For net1 and net2 layers you are using the same scope name, have you tried to change it accordingly to the rest of your layers? $\endgroup$
    – ignatius
    Jul 5, 2019 at 11:18
  • $\begingroup$ @ignatius, thank you for pointing it out. I will rename it but it seems tensorflow automatically renames it when the same name is already used as the documentation shows here: tensorflow.org/api_docs/python/tf/variable_scope $\endgroup$
    – smttsp
    Jul 5, 2019 at 11:51
  • $\begingroup$ What are the values of first_ac and second_ac? In comparison, you use 512 and 256 (i.e. hard coded numbers) for the classification head at the corresponding places. $\endgroup$
    – HelloGoodbye
    Jul 6, 2019 at 10:59
  • $\begingroup$ Also, you define both loss and loss2. Is the latter ever used? What is your objective function, i.e. what function does your optimizer try to minimize? $\endgroup$
    – HelloGoodbye
    Jul 6, 2019 at 11:10
  • $\begingroup$ @HelloGoodbye, I forgot to replace first_fc = 512 and second_fc=256. This code was a piece of code from hyperparameter tuning. My loss function is opt = tf.train.AdamOptimizer(learning_rate=lr).minimize(loss+0.2*loss2). 0.2 is just to normalize as approx. loss = loss2*0.2. You can see that subplot13 is converging to 18.5 whereas subplot12 is to 3.8. approx 5 times higher. I don't know if this was anyhow useful. $\endgroup$
    – smttsp
    Jul 8, 2019 at 11:21

1 Answer 1

Reset to default
1
$\begingroup$

You are using accuracy to estimate the performance of your model, which is a bad idea. You can't understand the limitations and advantages of your model using accuracy as a evaluation metric. Consider the following situations

  1. Your model predicts 36 instead of 37
  2. Your model predicts 106 instead of 37

Accuracy as a evaluation metric, treats both of them in the same way. I'd suggest you to go with MAE or RMSE for evaluation. You can train the network either using classification or regression, but make sure that you evaluate it using the right metric.

Btw, the multihead idea is really good.

Improve this answer
$\endgroup$
1
  • $\begingroup$ You are right about it. I have done multiple evaluations considering many different factors. Another criteria I have was MAE that I keep track of. For double head the MAE is 1.1 whereas for single head the error was around 2-3 if I remember correct. Regarding the multi-head, I just got lucky but I don't know why. Just trying to find out why $\endgroup$
    – smttsp
    Jul 5, 2019 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.