I am working on a research problem where I got stuck on something which doesn't really make sense to me. To explain the issue I will give an example of a similar problem (face age detection
).
Suppose that I have many face images that I precisely know how many **days** old
the person in each image. (I need precision in my problem).
I train my model using CNN in three ways.
In the first training, I create a neural network that gives an output of 75 classes (say the ages are 0-74) which tells me the correct age with 30-40%
accuracy. Below is the final code piece after the CNN layers:
net1 = tf.reduce_mean(net, axis=(1, 2), name='gap')
# Fully-connected classifier
net1 = slim.fully_connected(net1, 512, activation_fn=utils.lrelu, scope='cmi_ff1')
net1 = slim.dropout(net1, prob, scope='dropout1_net1')
net1 = slim.fully_connected(net1, 256, activation_fn=utils.lrelu, scope='cmi_ff2_net1')
net1 = slim.dropout(net1, prob, scope='dropout2_net1')
y_ = slim.fully_connected(net1, n_classes, activation_fn=tf.nn.softmax, scope='cmi_ffo_net1')
loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits( logits=y_, labels=y))
n_classes=75
as the number of ages is 75.
I keep everything the same and change the output layer from classification to a regression which results in significantly lower accuracy that each person approx 37 years old. This tells me that my neural network is trying to minimize the error and doesn't learn anything. I keep everything the same except for the output layer
and loss function
.
y2_ = slim.fully_connected(net1, 1, activation_fn=None, scope='cmi_ffo_net2')
loss2 = tf.reduce_mean(tf.abs(y2_ - tf.cast(y,tf.float32)))
Up to now, nothing sounds weird to me (although I am a newbie in DL). The weird thing is happening when I decide to use two heads instead of a single head. I split the network into two right after the last convolutional layer.
net1 = tf.reduce_mean(net, axis=(1, 2), name='gap')
net2 = tf.reduce_mean(net, axis=(1, 2), name='gap')
# Fully-connected classifier
net1 = slim.fully_connected(net1, 512, activation_fn=activation, scope='cmi_ff1')
net1 = slim.dropout(net1, prob, scope='dropout_net1')
net1 = slim.fully_connected(net1, 256, activation_fn=activation, scope='cmi_ff2_net1')
net1 = slim.dropout(net1, prob, scope='dropout_net1')
y_ = slim.fully_connected(net1, n_classes, activation_fn=tf.nn.softmax, scope='cmi_ffo_net1')
loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y_, labels=y))
# Fully-connected regressor
net2 = slim.fully_connected(net2, first_fc, activation_fn=activation, scope='cmi_ff1')
net2 = slim.dropout(net2, prob, scope='dropout_net1')
net2 = slim.fully_connected(net2, second_fc, activation_fn=activation, scope='cmi_ff2_net2')
y2_ = slim.fully_connected(net2, 1, activation_fn=None, scope='cmi_ffo_net2')
loss2 = tf.reduce_mean(tf.abs(y2_ - tf.cast(y,tf.float32)))
Here, the accuracy of classification goes up to 65-70%
and regression remains to the same (converges to the middle value). While regression seems to be useless in the entire process, somehow it gives an amazing push to the classification to improve accuracy that I can't even speculate about the reason.
I am adding a figure that shows the accuracy and loss of the two-headed network. (I was trying to optimize some hyperparameters.)
Explanation of the figure:
subplot 1,1: Training accuracy for classification head
subplot 2,1: Validation accuracy for classification head
subplot 1,2: Loss for classification head
subplot 1,3: Loss for regression head
subplot 2,2: Total loss: 0.2* regression + classification
I wonder what could be the reason for this or is there something that I do wrong? Also, I know the code is not complete but I can't share the entire code. I just need some directions about what to think and where to look.
tensorflow
automatically renames it when the same name is already used as the documentation shows here: tensorflow.org/api_docs/python/tf/variable_scope $\endgroup$first_ac
andsecond_ac
? In comparison, you use512
and256
(i.e. hard coded numbers) for the classification head at the corresponding places. $\endgroup$loss
andloss2
. Is the latter ever used? What is your objective function, i.e. what function does your optimizer try to minimize? $\endgroup$first_fc = 512
andsecond_fc=256
. This code was a piece of code fromhyperparameter tuning
. My loss function isopt = tf.train.AdamOptimizer(learning_rate=lr).minimize(loss+0.2*loss2)
.0.2
is just to normalize as approx.loss = loss2*0.2
. You can see thatsubplot13
is converging to18.5
whereassubplot12
is to3.8
. approx 5 times higher. I don't know if this was anyhow useful. $\endgroup$