1
$\begingroup$

I trained a CNN based model on the Stanford Car Dataset and got 99% accuracy. I used test dataset to evaluate the model but it performed very poorly. I knew my model was overfitting and I tried to reduce that, but that's for later. The thing is that I got curious and just to see it predict something correctly, I ran the model on the "training set".

Now, this is where everything went weird. No matter how much the model is overfitting, it was supposed to give correct answers (atleast 99% of times) on training data. But it gave almost all wrong answers. I rerun the whole model, checked if there were some issues and I just can't seem to figure out what is wrong.

Using Image Data Generator

IMG_SIZE = 256
BATCH_SIZE = 16

datagen = ImageDataGenerator(
    rescale=1./255,
    validation_split=0.1)

train_generator = datagen.flow_from_dataframe(
    dataframe = df, 
    directory = "/home/ashok/Downloads/cars_train", 
    x_col = "filename", 
    y_col = "class", 
    class_mode = "categorical", 
    target_size = (IMG_SIZE,IMG_SIZE), 
    subset='training',
    batch_size = BATCH_SIZE)

valid_generator = datagen.flow_from_dataframe(
    dataframe = df, 
    directory = "/home/ashok/Downloads/cars_train", 
    x_col = "filename", 
    y_col = "class", 
    class_mode = "categorical", 
    target_size = (IMG_SIZE,IMG_SIZE), 
    subset='validation',
    batch_size = BATCH_SIZE)

Base Model

IMG_SHAPE = (IMG_SIZE, IMG_SIZE, 3)

base_model = tf.keras.applications.ResNet50(input_shape=IMG_SHAPE,
                                               include_top=False,
                                               weights='imagenet')
base_model.trainable = False

Adding to Base Model

model = keras.Sequential([
    base_model,
    keras.layers.MaxPool2D(2,2),
    keras.layers.Flatten(),
    keras.layers.Dense(512, activation = 'relu'),
    keras.layers.Dense(196, activation = 'softmax')
])

Compiling and Fitting

model.compile(Adam(lr=0.0001),
              loss="categorical_crossentropy",
              metrics=["accuracy"])

history = model.fit_generator(generator=train_generator,
                    validation_data=valid_generator,
                    epochs=10)

Last Epoch

Epoch 10/10
459/459 [==============================] - 111s 242ms/step - loss: 0.0468 - accuracy: 0.9958 - val_loss: 6.5822 - val_accuracy: 0.0061

Testing on Training Data

test_datagen = ImageDataGenerator(rescale=1. / 255)

test_generator = test_datagen.flow_from_directory(
    directory="/home/ashok/Downloads/Train",
    target_size=(IMG_SIZE, IMG_SIZE),
    batch_size=BATCH_SIZE,
    class_mode=None,
    shuffle=False
)

test_generator.reset()

predicted_class_indices = np.argmax(pred,axis=1)
print(max(predicted_class_indices)) ## Also one interesting thing is that it doesnt even predict any class below 39 (min) or above 159(max) (there are total 196 classes). 

Can anybody help me with these issues? Am I missing something important?

PS: Also I am curious as to why my Val_acc doesn't increase at all, even though I tried training with data augmentation too. it shouldn't be that bad in any case.

$\endgroup$
4
  • $\begingroup$ About testing on training data - did you try using the exact same generator that you used for training? Your code shows that it's not exactly the same. $\endgroup$ Dec 24, 2019 at 8:46
  • $\begingroup$ Agree. You are using different directories in the generator. Check for it. $\endgroup$
    – chzhrr
    Dec 24, 2019 at 8:53
  • $\begingroup$ @chzhrr, I did not use the exact same generator because the first one has the info about classes read from a dataframe df. So created a new generator using the exact same data. I just moved the training data inside a directory named Train so it can detect the images as its required for using flow_from_directory. The data is the same in both cases. $\endgroup$ Dec 24, 2019 at 9:47
  • $\begingroup$ @ItamarMushkin Its not exactly the same, but it does have the same data in both cases as explained in another comment. Is it supposed to be exactly same in every sense? I tried reading a Single Image for prediction also (using CV2) from the training dataset, just to be sure. It always gives wrong prediction in that case too. $\endgroup$ Dec 24, 2019 at 9:51

1 Answer 1

1
$\begingroup$

There are many possible errors in your code:

  • As mentioned in comments, there could be errors in your directories.
  • It is unclear how pred variable is created.
  • You are unnecessarily implementing test accuracy. It would be better to use Keras' built-in methods, in particular model.evaluate()
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.