0
$\begingroup$

I have a training dataset of 10000 pictures and a test dataset of 15000 pictures. There are 23 types of birds.

First of all, I imported the necessary

import tensorflow as tf 
from tensorflow import keras
from tensorflow.keras.preprocessing.image import ImageDataGenerator 
from tensorflow.keras import layers 
from tensorflow.keras import Model 
import matplotlib.pyplot as plt

from keras.applications.inception_v3 import InceptionV3, preprocess_input

batch_size = 32
IM_WIDTH, IM_HEIGHT = 150, 150 # fixed size for inceptionV3
nb_epochs = 13

train_dir = '/kaggle/output/working_directory/'

I am using ImageDataGenerator for Image augmentation

#test_datagen = ImageDataGenerator(rescale = 1.0/255.)
test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)

train_datagen = ImageDataGenerator(
            preprocessing_function=preprocess_input,
            rotation_range = 40, 
            width_shift_range = 0.2, 
            height_shift_range = 0.2,
            shear_range = 0.2, 
            zoom_range = 0.2, 
            horizontal_flip = True,
            validation_split=0.2) # set validation split

And importing data using flow_from_directory

train_generator = train_datagen.flow_from_directory(train_dir, 
                                                    batch_size = batch_size, 
                                                    class_mode = 'categorical', 
                                                    target_size = (IM_WIDTH, IM_HEIGHT),
                                                    shuffle=True,
                                                    subset='training')

validation_generator = train_datagen.flow_from_directory(train_dir, 
                                                              batch_size = batch_size, 
                                                              class_mode = 'categorical', 
                                                              target_size = (IM_WIDTH, IM_HEIGHT),
                                                              shuffle=True,
                                                              subset='validation')

test_generator = test_datagen.flow_from_directory(
    directory = '/kaggle/input/test/',
    target_size = (IM_WIDTH, IM_HEIGHT),
    color_mode = 'rgb',
    batch_size = 1,
    class_mode = None,
    shuffle = False)

Found 8225 images belonging to 23 classes.

Found 2045 images belonging to 23 classes.

Found 15009 images belonging to 1 classes.

Finally, I imported the actual model

from tensorflow.keras.applications.inception_v3 import InceptionV3
base_model = InceptionV3(input_shape = (IM_WIDTH, IM_HEIGHT, 3), include_top = False, weights = 'imagenet')

for layer in base_model.layers:
    layer.trainable = True

import keras
from tensorflow.keras.optimizers import RMSprop

x = layers.Flatten()(base_model.output)
x = layers.Dense(1024, activation='relu')(x)
x = layers.Dropout(0.4)(x)
x = layers.Dense(23, activation='softmax')(x)

model = tf.keras.models.Model(base_model.input, x)

model.compile(optimizer = keras.optimizers.Adam(lr=0.0001), loss = 'categorical_crossentropy', metrics = ['acc'])

from keras.callbacks import ModelCheckpoint
from keras.callbacks import EarlyStopping

filepath = 'best_model.h5'

es = EarlyStopping(monitor='val_acc', 
                   mode='max', 
                   verbose=1, 
                   patience=3)

ModelCheckpoint = ModelCheckpoint(filepath,
                             monitor='val_acc',
                             mode='max',
                             save_best_only=True,
                             verbose=1)

callbacks_list = [ModelCheckpoint, es]

inception = model.fit(train_generator, 
                      steps_per_epoch = train_generator.samples // batch_size,
                      validation_data = validation_generator,
                      validation_steps = validation_generator.samples// batch_size,
                      epochs = nb_epochs,
                      callbacks = callbacks_list)

Epoch 00012: val_acc did not improve from 0.86210 Epoch 13/13 257/257 [==============================] - 91s 355ms/step - loss: 0.2282 - acc: 0.9288 - val_loss: 0.5141 - val_acc: 0.8676

Epoch 00013: val_acc improved from 0.86210 to 0.86756, saving model to best_model.h5

Now, testing:

from keras.models import load_model

model = load_model('best_model.h5')

test_generator.reset()
STEP_SIZE_TEST=test_generator.n//test_generator.batch_size


y_pred = model.predict(test_generator,
                       steps = STEP_SIZE_TEST)

predictions = [np.argmax(pred) for pred in y_pred]

prediction = pd.DataFrame(predictions, columns=['label']).to_csv('prediction.csv')
df.to_csv(index=True)

After I submit the .cvs file, the accuracy is 4.5%. I am very confused as validation data returns approx. 85% and it is not compromised, the model is not training on validation data. Hence, I am very confused why does my model achieve only 4.5% on the testing dataset. I believe there is something wrong with .prediction and storing the predicted values, but I cannot figure it out.

$\endgroup$
8
  • $\begingroup$ I guess that your model has been trained on a set of data, and tested on a completely different distribution. What I mean is that validation and test set should have the same distribution. Think about the following example: you train a model that classifies images of cat vs dog. Peformances: train_acc: 92%, val_acc 90%. Now your test set is made up just by horses images, do you think your model will get a nice test accuracy? $\endgroup$ – Oscar Apr 12 at 10:31
  • $\begingroup$ @Oscar Hi Oscar, Thanks for your comment. That is exactly what I have thought of in the first place. However, this project's got a leaderboard and there are models with very high test accuracy, +90%. Therefore, I am sure that my code's got a bug that brings the accuracy down or makes the .csv document inaccurate. $\endgroup$ – Trixiew Apr 12 at 10:50
  • $\begingroup$ actually looking at the code, I think your model does train on validation data. You use a generator, you can have a look at Keras documentation, at each epoch it might happen that the validation images taken by validation_generator are overlapped with the training_generator ones. $\endgroup$ – Oscar Apr 12 at 12:43
  • $\begingroup$ @Oscar but I am also using validation_split = 0.2, and after subset training and validation. I was following keras documentation and that is how I got it, the data should be separated for training and validation, isn't it? $\endgroup$ – Trixiew Apr 12 at 17:19
  • 1
    $\begingroup$ You are using the same dir as train and val. Please use "val split" and share the result $\endgroup$ – 10xAI Apr 16 at 16:12
0
$\begingroup$

I believe this could help someone. The problem was that the output classes were randomly assigned. My classes are called: 0,1,2,3,4...,22. However, DataGenerator assigned output '5' to class 13, output '7' to class 15, and so on. Hence, the classes were shuffled. It is important to assign the output to each class.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.