# Testing accuracy very low, while training and validation accuracy ~ 85%

I have a training dataset of 10000 pictures and a test dataset of 15000 pictures. There are 23 types of birds.

First of all, I imported the necessary

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras import layers
from tensorflow.keras import Model
import matplotlib.pyplot as plt

from keras.applications.inception_v3 import InceptionV3, preprocess_input

batch_size = 32
IM_WIDTH, IM_HEIGHT = 150, 150 # fixed size for inceptionV3
nb_epochs = 13

train_dir = '/kaggle/output/working_directory/'


I am using ImageDataGenerator for Image augmentation

#test_datagen = ImageDataGenerator(rescale = 1.0/255.)
test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)

train_datagen = ImageDataGenerator(
preprocessing_function=preprocess_input,
rotation_range = 40,
width_shift_range = 0.2,
height_shift_range = 0.2,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True,
validation_split=0.2) # set validation split


And importing data using flow_from_directory

train_generator = train_datagen.flow_from_directory(train_dir,
batch_size = batch_size,
class_mode = 'categorical',
target_size = (IM_WIDTH, IM_HEIGHT),
shuffle=True,
subset='training')

validation_generator = train_datagen.flow_from_directory(train_dir,
batch_size = batch_size,
class_mode = 'categorical',
target_size = (IM_WIDTH, IM_HEIGHT),
shuffle=True,
subset='validation')

test_generator = test_datagen.flow_from_directory(
directory = '/kaggle/input/test/',
target_size = (IM_WIDTH, IM_HEIGHT),
color_mode = 'rgb',
batch_size = 1,
class_mode = None,
shuffle = False)


Found 8225 images belonging to 23 classes.

Found 2045 images belonging to 23 classes.

Found 15009 images belonging to 1 classes.

Finally, I imported the actual model

from tensorflow.keras.applications.inception_v3 import InceptionV3
base_model = InceptionV3(input_shape = (IM_WIDTH, IM_HEIGHT, 3), include_top = False, weights = 'imagenet')

for layer in base_model.layers:
layer.trainable = True

import keras
from tensorflow.keras.optimizers import RMSprop

x = layers.Flatten()(base_model.output)
x = layers.Dense(1024, activation='relu')(x)
x = layers.Dropout(0.4)(x)
x = layers.Dense(23, activation='softmax')(x)

model = tf.keras.models.Model(base_model.input, x)

model.compile(optimizer = keras.optimizers.Adam(lr=0.0001), loss = 'categorical_crossentropy', metrics = ['acc'])

from keras.callbacks import ModelCheckpoint
from keras.callbacks import EarlyStopping

filepath = 'best_model.h5'

es = EarlyStopping(monitor='val_acc',
mode='max',
verbose=1,
patience=3)

ModelCheckpoint = ModelCheckpoint(filepath,
monitor='val_acc',
mode='max',
save_best_only=True,
verbose=1)

callbacks_list = [ModelCheckpoint, es]

inception = model.fit(train_generator,
steps_per_epoch = train_generator.samples // batch_size,
validation_data = validation_generator,
validation_steps = validation_generator.samples// batch_size,
epochs = nb_epochs,
callbacks = callbacks_list)


Epoch 00012: val_acc did not improve from 0.86210 Epoch 13/13 257/257 [==============================] - 91s 355ms/step - loss: 0.2282 - acc: 0.9288 - val_loss: 0.5141 - val_acc: 0.8676

Epoch 00013: val_acc improved from 0.86210 to 0.86756, saving model to best_model.h5

Now, testing:

from keras.models import load_model

test_generator.reset()
STEP_SIZE_TEST=test_generator.n//test_generator.batch_size

y_pred = model.predict(test_generator,
steps = STEP_SIZE_TEST)

predictions = [np.argmax(pred) for pred in y_pred]

prediction = pd.DataFrame(predictions, columns=['label']).to_csv('prediction.csv')
df.to_csv(index=True)


After I submit the .cvs file, the accuracy is 4.5%. I am very confused as validation data returns approx. 85% and it is not compromised, the model is not training on validation data. Hence, I am very confused why does my model achieve only 4.5% on the testing dataset. I believe there is something wrong with .prediction and storing the predicted values, but I cannot figure it out.

• I guess that your model has been trained on a set of data, and tested on a completely different distribution. What I mean is that validation and test set should have the same distribution. Think about the following example: you train a model that classifies images of cat vs dog. Peformances: train_acc: 92%, val_acc 90%. Now your test set is made up just by horses images, do you think your model will get a nice test accuracy? – Oscar Apr 12 at 10:31
• @Oscar Hi Oscar, Thanks for your comment. That is exactly what I have thought of in the first place. However, this project's got a leaderboard and there are models with very high test accuracy, +90%. Therefore, I am sure that my code's got a bug that brings the accuracy down or makes the .csv document inaccurate. – Trixiew Apr 12 at 10:50
• actually looking at the code, I think your model does train on validation data. You use a generator, you can have a look at Keras documentation, at each epoch it might happen that the validation images taken by validation_generator are overlapped with the training_generator ones. – Oscar Apr 12 at 12:43
• @Oscar but I am also using validation_split = 0.2, and after subset training and validation. I was following keras documentation and that is how I got it, the data should be separated for training and validation, isn't it? – Trixiew Apr 12 at 17:19
• You are using the same dir as train and val. Please use "val split" and share the result – 10xAI Apr 16 at 16:12