# VGG16 based model not learning to recognize emotions from videos

My model looks like this

model = keras.Sequential(
[
layers.MaxPooling2D(),
layers.MaxPooling2D(),
layers.MaxPooling2D(),
layers.MaxPooling2D(),
layers.MaxPooling2D(),
layers.Flatten(),
layers.Dense(4096, activation='relu'),
layers.Dropout(0.5),
layers.Dense(4096, activation='relu'),
layers.Dropout(0.5),
layers.Dense(2622, activation='relu'),
layers.Dropout(0.5),
layers.Dense(1, activation='linear')
]
)


My training set is made of almost 1mil color 96x96 images showing faces. The target is one value for each image. I'm trying to predict the valence of the emotion, that is how positive or negative an emotion is. This value can be any float in [-1,1]. The input images are normalized in the same range [-1,1].

For validation, I'm using almost 50k images with the same characteristics as the training set. Training and validation sets are completely separate.

This is how I'm training the model

learning_schedule = ExponentialDecay(
initial_learning_rate=1e-5,
decay_steps=400,
decay_rate=0.97)
model.compile(optimizer=opt,
loss=CCC,
metrics=['mse'])

print("[INFO] training model...")
history = model.fit(ds,
validation_data=(testImages, testLabels),
epochs=5, shuffle=False)


CCC is a custom loss function (Lin's Concordance Correlation Coefficient)

def CCC(y_true, y_pred):

import keras.backend as K
# covariance between y_true and y_pred
s_xy = K.mean( (y_true - K.mean(y_true)) * (y_pred - K.mean(y_pred)) )
# means
x_m = K.mean(y_true)
y_m = K.mean(y_pred)
# variances
s_x_sq = K.var(y_true)
s_y_sq = K.var(y_pred)

# condordance correlation coefficient
ccc = (2.0*s_xy) / (s_x_sq + s_y_sq + (x_m-y_m)**2)

return 1 - ccc


Finally I'm using batch_size=320. The data in ds is shuffled by chunks of 80 because the CCC loss function is very small (even 0) if the valence values are similar. Since the images are taken from videos, it's very likely that consecutive frames show similar valence. As I said, to prevent this I split the input into chunks of 80 images (frames) and I shuffle these chunks. So batch_size=320 means that each batch is made of 4 chunks of 80 images and these 4 chunks are likely from different videos as they have been shuffled beforehand. This is a requirement of my model as later I will need to add a copule of GRU layers which require a sequence.

This is the loss after 5 epochs

This is the MSE after 5 epochs

I actually have tested the model for up to 50 epochs and the plot is the same. The train loss decreases as it should but the valid loss doesen't move. Valid loss is actually not exactly 1 but it decreases to just about 0.998ish, and it stays like that forever. I thought the loss function could be the problem and I switched to MSE, but I got similar results. I've tried with a smaller learning rate and with a fixed learning rate. I'va also tried to increase the drop probability of dropouts. All without success.

The model architecture is quite large but the dataset is very big too. I don't undertand why it's not learning. Do you have any good suggestions?

• it seems to overfit Feb 1 at 19:16
• Yes I noticed that too. So I over simplified the architecture leaving only the first conv, the max pool, the flatten and the last linear dense. It was very surprising to see that the train loss still managed to go down similarly to the plot above whereas the valid loss stayed close to 1. How can a model so small learn so quickly on a train dataset with 1mil examples but is not able to generalize? I can't get my head around it.
– zcb
Feb 1 at 19:34
• 1. Each batch has 4 chunks of 80 images, and within each chunk, the images are likely very similar? That sounds like it could be a problem. Instead, I recommend you use 320 frames taken from random positions in 320 randomly chosen videos (i.e., typically it will be 320 different videos, not 4). 2. Do you have any evidence that it is possible to solve the task? Have you found any baseline or prior work that solves the task? How well can you do as a human?
– D.W.
Feb 7 at 6:51
• 3. CCC is a weird loss function. Why are you using that? Have you tried using a standard loss function, like MSE loss? That might be a very informative experiment. 4. I didn't understand your comments about GRU layers and a sequence. For this task (classify one image), I suggest you first try to solve this task in the most natural way. If you later want to classify sequences, do that with a model designed for sequences, not this model.
– D.W.
Feb 7 at 6:53
• I know it's a weired function but I'm using it because it's used in the original paper. Basically I'm just trying to reproduce their results. Using MSE as a loss function seems to help, so I think I'll give up on ccc. In case you're interested here's the link to the paper: researchgate.net/publication/…
– zcb
Feb 7 at 10:00