# Why does “classification+regression” work better than “classification” in my NN?

I am working on a research problem where I got stuck on something which doesn't really make sense to me. To explain the issue I will give an example of a similar problem (face age detection).

Suppose that I have many face images that I precisely know how many **days** old the person in each image. (I need precision in my problem).

I train my model using CNN in three ways.

In the first training, I create a neural network that gives an output of 75 classes (say the ages are 0-74) which tells me the correct age with 30-40% accuracy. Below is the final code piece after the CNN layers:

net1 = tf.reduce_mean(net, axis=(1, 2), name='gap')

# Fully-connected classifier
net1 = slim.fully_connected(net1, 512, activation_fn=utils.lrelu, scope='cmi_ff1')
net1 = slim.dropout(net1, prob, scope='dropout1_net1')
net1 = slim.fully_connected(net1, 256, activation_fn=utils.lrelu, scope='cmi_ff2_net1')
net1 = slim.dropout(net1, prob, scope='dropout2_net1')

y_ = slim.fully_connected(net1, n_classes, activation_fn=tf.nn.softmax, scope='cmi_ffo_net1')
loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits( logits=y_, labels=y))


n_classes=75 as the number of ages is 75.

I keep everything the same and change the output layer from classification to a regression which results in significantly lower accuracy that each person approx 37 years old. This tells me that my neural network is trying to minimize the error and doesn't learn anything. I keep everything the same except for the output layer and loss function.

y2_ = slim.fully_connected(net1, 1, activation_fn=None, scope='cmi_ffo_net2')
loss2 = tf.reduce_mean(tf.abs(y2_ - tf.cast(y,tf.float32)))


Up to now, nothing sounds weird to me (although I am a newbie in DL). The weird thing is happening when I decide to use two heads instead of a single head. I split the network into two right after the last convolutional layer.

net1 = tf.reduce_mean(net, axis=(1, 2), name='gap')
net2 = tf.reduce_mean(net, axis=(1, 2), name='gap')

# Fully-connected classifier
net1 = slim.fully_connected(net1, 512, activation_fn=activation, scope='cmi_ff1')
net1 = slim.dropout(net1, prob, scope='dropout_net1')
net1 = slim.fully_connected(net1, 256, activation_fn=activation, scope='cmi_ff2_net1')
net1 = slim.dropout(net1, prob, scope='dropout_net1')

y_ = slim.fully_connected(net1, n_classes, activation_fn=tf.nn.softmax, scope='cmi_ffo_net1')
loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y_, labels=y))

# Fully-connected regressor
net2 = slim.fully_connected(net2, first_fc, activation_fn=activation, scope='cmi_ff1')
net2 = slim.dropout(net2, prob, scope='dropout_net1')
net2 = slim.fully_connected(net2, second_fc, activation_fn=activation, scope='cmi_ff2_net2')

y2_ = slim.fully_connected(net2, 1, activation_fn=None, scope='cmi_ffo_net2')
loss2 = tf.reduce_mean(tf.abs(y2_ - tf.cast(y,tf.float32)))


Here, the accuracy of classification goes up to 65-70% and regression remains to the same (converges to the middle value). While regression seems to be useless in the entire process, somehow it gives an amazing push to the classification to improve accuracy that I can't even speculate about the reason.

I am adding a figure that shows the accuracy and loss of the two-headed network. (I was trying to optimize some hyperparameters.)

Explanation of the figure:

subplot 1,1: Training accuracy for classification head
subplot 2,1: Validation accuracy for classification head
subplot 1,2: Loss for classification head
subplot 1,3: Loss for regression head
subplot 2,2: Total loss: 0.2* regression + classification


I wonder what could be the reason for this or is there something that I do wrong? Also, I know the code is not complete but I can't share the entire code. I just need some directions about what to think and where to look.

• For net1 and net2 layers you are using the same scope name, have you tried to change it accordingly to the rest of your layers? – ignatius Jul 5 '19 at 11:18
• @ignatius, thank you for pointing it out. I will rename it but it seems tensorflow automatically renames it when the same name is already used as the documentation shows here: tensorflow.org/api_docs/python/tf/variable_scope – smttsp Jul 5 '19 at 11:51
• What are the values of first_ac and second_ac? In comparison, you use 512 and 256 (i.e. hard coded numbers) for the classification head at the corresponding places. – HelloGoodbye Jul 6 '19 at 10:59
• Also, you define both loss and loss2. Is the latter ever used? What is your objective function, i.e. what function does your optimizer try to minimize? – HelloGoodbye Jul 6 '19 at 11:10
• @HelloGoodbye, I forgot to replace first_fc = 512 and second_fc=256. This code was a piece of code from hyperparameter tuning. My loss function is opt = tf.train.AdamOptimizer(learning_rate=lr).minimize(loss+0.2*loss2). 0.2 is just to normalize as approx. loss = loss2*0.2. You can see that subplot13 is converging to 18.5 whereas subplot12 is to 3.8. approx 5 times higher. I don't know if this was anyhow useful. – smttsp Jul 8 '19 at 11:21

• You are right about it. I have done multiple evaluations considering many different factors. Another criteria I have was MAE that I keep track of. For double head the MAE is 1.1 whereas for single head the error was around 2-3 if I remember correct. Regarding the multi-head, I just got lucky but I don't know why. Just trying to find out why – smttsp Jul 5 '19 at 12:05