I trained a CNN based model on the Stanford Car Dataset and got 99% accuracy. I used test dataset to evaluate the model but it performed very poorly. I knew my model was overfitting and I tried to reduce that, but that's for later. The thing is that I got curious and just to see it predict something correctly, I ran the model on the "training set".

Now, this is where everything went weird. No matter how much the model is overfitting, it was supposed to give correct answers (atleast 99% of times) on training data. But it gave almost all wrong answers. I rerun the whole model, checked if there were some issues and I just can't seem to figure out what is wrong.

Using Image Data Generator

IMG_SIZE = 256
BATCH_SIZE = 16

datagen = ImageDataGenerator(
    rescale=1./255,
    validation_split=0.1)

train_generator = datagen.flow_from_dataframe(
    dataframe = df, 
    directory = "/home/ashok/Downloads/cars_train", 
    x_col = "filename", 
    y_col = "class", 
    class_mode = "categorical", 
    target_size = (IMG_SIZE,IMG_SIZE), 
    subset='training',
    batch_size = BATCH_SIZE)

valid_generator = datagen.flow_from_dataframe(
    dataframe = df, 
    directory = "/home/ashok/Downloads/cars_train", 
    x_col = "filename", 
    y_col = "class", 
    class_mode = "categorical", 
    target_size = (IMG_SIZE,IMG_SIZE), 
    subset='validation',
    batch_size = BATCH_SIZE)

Base Model

IMG_SHAPE = (IMG_SIZE, IMG_SIZE, 3)

base_model = tf.keras.applications.ResNet50(input_shape=IMG_SHAPE,
                                               include_top=False,
                                               weights='imagenet')
base_model.trainable = False

Adding to Base Model

model = keras.Sequential([
    base_model,
    keras.layers.MaxPool2D(2,2),
    keras.layers.Flatten(),
    keras.layers.Dense(512, activation = 'relu'),
    keras.layers.Dense(196, activation = 'softmax')
])

Compiling and Fitting

model.compile(Adam(lr=0.0001),
              loss="categorical_crossentropy",
              metrics=["accuracy"])

history = model.fit_generator(generator=train_generator,
                    validation_data=valid_generator,
                    epochs=10)

Last Epoch

Epoch 10/10
459/459 [==============================] - 111s 242ms/step - loss: 0.0468 - accuracy: 0.9958 - val_loss: 6.5822 - val_accuracy: 0.0061

Testing on Training Data

test_datagen = ImageDataGenerator(rescale=1. / 255)

test_generator = test_datagen.flow_from_directory(
    directory="/home/ashok/Downloads/Train",
    target_size=(IMG_SIZE, IMG_SIZE),
    batch_size=BATCH_SIZE,
    class_mode=None,
    shuffle=False
)

test_generator.reset()

predicted_class_indices = np.argmax(pred,axis=1)
print(max(predicted_class_indices)) ## Also one interesting thing is that it doesnt even predict any class below 39 (min) or above 159(max) (there are total 196 classes). 

Can anybody help me with these issues? Am I missing something important?

PS: Also I am curious as to why my Val_acc doesn't increase at all, even though I tried training with data augmentation too. it shouldn't be that bad in any case.