I tried to develop a number of CNN architectures to train on a 1000-point subset of the "cat-dog" Kaggle training set (meaning, by the way, that all 1000 data points were labeled). I used a 700-150-150 train-validate-test split, and used the following code on an Xception architecture:

pre_xception_model = keras.applications.Xception(include_top=False, weights='imagenet',
                                                 input_tensor=None, input_shape=(224,224,3), pooling=None, classes=2)
for layer in pre_xception_model.layers:
    layer.trainable = False
dropout = Dropout(0.5)(pre_xception_model.output)
flatten = Flatten()(dropout)   
output = Dense(2, activation='softmax')(flatten)
xception_model = Model(pre_xception_model.input, output)

xception_model.compile(Adam(lr=.0001, decay=1e-6), loss='categorical_crossentropy', metrics=['accuracy'])

aug = ImageDataGenerator(rotation_range=20, zoom_range=0.15,
    width_shift_range=0.2, height_shift_range=0.2, shear_range=0.15,
    horizontal_flip=True, fill_mode="nearest")
batches = 20

xception_model.fit_generator(aug.flow(X_train, y_train, batch_size = batches), steps_per_epoch = len(X_train) // batches,
                          validation_data = (X_valid, y_valid), validation_steps = 4, epochs = 10, verbose = 1)

xception_prob = xception_model.predict(X_test, verbose=1)
xception_predict = xception_prob.argmax(axis=-1)
cm_xception = confusion_matrix(y_test[:,1], xception_predict)
plot_confusion_matrix(cm_xception, cm_plot_labels, title='Confusion Matrix')

print(f'\nAccuracy = {(cm_xception[0,0]+cm_xception[1,1])/150}\n')

This produced this training log, and the .predict() produced this confusion matrix. I'm at a loss as to explain the overfitting; can someone here help me as to where I went wrong on this?

  • 1
    $\begingroup$ It's bit difficult for me to understand from the code. But possible over-fitting reason in your case is, "you dont have sufficient cat images" for training. If you have good average ratio of both types, possibly cat images moved in to CV and Test partition and thats the reason your model is over-fitted with Dogs image. $\endgroup$ Jun 4 '19 at 17:46
  • $\begingroup$ What I find strange about that is that I trained a ResNet model on the same data, which resulted in 0.96 accuracy. $\endgroup$
    – Yehuda
    Jun 4 '19 at 17:49
  • $\begingroup$ According to me, above mentioned theory is still applicable for ResNet. $\endgroup$ Jun 4 '19 at 17:51

Correct me if I'm wrong, but it seems you have a binary classification problem - the image either contains a cat or a dog.

If that's the case, then:

output = Dense(1, activation='sigmoid')(flatten)


xception_model.compile(Adam(lr=.0001, decay=1e-6), loss='binary_crossentropy', metrics=['binary_accuracy'])

This thread is a great resource on the difference between the two questions.

Let me know if it works.

EDIT: Further explanation

Although we normally talk about “binary classification”, the way the outcome is usually modeled is as a Bernoulli random variable, conditioned on the input data. So:

$$P(y = 1|\mathbf{x}) = p, \ 0\leq p\leq1$$

A Bernoulli random variable takes on values between 0 and 1. So that’s what our network should produce. One idea might be to just clip all values of $wth+b$ outside that interval. But if we do this, the gradient in these regions will be 0: The network cannot learn.

A better way is to squish the complete incoming interval into the range (0,1), using the sigmoid function:

$$\sigma(x) = \frac{1}{1 + e^{(-x)}}$$

But the sigmoid function saturates when its input gets very large, or very small. Is this problematic? It depends. In the end, what we care about is if the cost function saturates. If we follow the general principle of maximum likelihood/cross entropy, the loss will be:

$$ - log P (y|\mathbf{x}) $$

where the $log$ undoes the $exp$ in the sigmoid.

In Keras, the corresponding loss function is binary_crossentropy. For a single item, the loss will be:

  • $−log(p)$ when the ground truth is 1

  • $−log(1−p)$ when the ground truth is 0

Here, you can see that when for an individual example, the network predicts the wrong class and is highly confident about it, this example will contribute very strongly to the loss. Therefore, you only need to have one output node, because binary classification is "symmetrical". By that, I mean that if we say that an item is of class 0 with probability p, we are also referring to the fact that this same item is of class 1 with probability 1 - p.

  • $\begingroup$ If I may—why would the Dense layer have only one unit? Wouldn’t you need two for a binary classification? $\endgroup$
    – Yehuda
    Jun 4 '19 at 17:59
  • $\begingroup$ I updated my response with a more in-depth explanation, as well as answering this question. $\endgroup$ Jun 4 '19 at 18:26

Try changing your code as follows:

pre_xception_model = keras.applications.Xception(include_top=False, weights='imagenet',
                                                 input_tensor=None, input_shape=(224,224,3), pooling='avg', classes=2)
x=keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
output = Dense(2, activation='softmax')(x)
xception_model = Model(pre_xception_model.input, output)
for layer in pre_xception_model.layers:
    layer.trainable = True
xception_model.compile(Adam(lr=.0001, decay=1e-6), loss='categorical_crossentropy', metrics=['accuracy'])

This adds an average pooling layer to the model, then batch normalization and your final classification layer. I use this for the MobileNet model and it works well. I would also recommend you use the callbacks ModelCheckpoint and ReduceLROnPlateau. Documentation for these are here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.