Issue with predict generator keras

Question

I'm new with keras with tensorflow backend and I'm trying to do transfer learning with pretrained net. The problem is that the accuracy on validation set is very high, around the 90% , but on test set the accuracy is very bad, less that 1%. I solved the problem using opencv to read and resize image, but I'd like to understand why with keras methods I've this problem. I paste my code below.

from keras.preprocessing.image import ImageDataGenerator
from keras.applications.xception import preprocess_input
import keras

train_val_datagen = ImageDataGenerator( validation_split=0.25, preprocessing_function=preprocess_input)


train_val_generator = train_val_datagen.flow_from_directory( # subset di allenamento
    directory="./image-dataset/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=32,
    class_mode="categorical",
    shuffle=True,
    subset = 'training',
    seed=17)
val_train_generator = train_val_datagen.flow_from_directory( # subset di validation
    directory="./image-dataset/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=32,
    class_mode="categorical",
    shuffle=True,
    subset = 'validation',
    seed=17)
final_train_generator = train_val_datagen.flow_from_directory( # set finale di allenamento con tutti i dati
    directory="./image-dataset/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=32,
    class_mode="categorical",
    shuffle=True,
    seed=17)

As you can see i used Xception as pretrained-net and I choose to resize my images to adapt them to the net.

After the training I created a new Iterator for test data as follow:

test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)
test_generator = test_datagen.flow_from_directory( 
    directory="./TEST/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=1, # predico una alla volta
    shuffle = False,
    class_mode=None # non ce alcuna classe di riferimento
    )
test_generator.reset()

where the preprocess function is exactly the same.

The predictions are made with the following code:

predictions = model_xcpetion.predict_generator(test_generator, 6104, verbose = 1 )

where 6104 is the number of images in test folder. After this i've generated a csv with images name with relative categorical probabilities:

import pandas as pd
import numpy as np
df = pd.DataFrame(predictions)
cols =[('probability of' + str(i)) for i in list(range(1, 30 )) ]
df.columns =  cols
df['images'] = imNames
df.to_csv('predictions_xception_all_data.csv', sep=',') 

where the columns represent the labels (1 to 29) and imNames are obtained with filenames attribute of test_generator. Finally I generated the csv with the labels with highest probabilities value and i compute the accuracy obtaining the value that I wrote before.

The code that I used to solve is the same but I read and resize images with the following code:

width = 299
height = 299
dim = (width, height)
images = []
# for each img
resized = cv2.resize(img, dim, interpolation = cv2.INTER_AREA)
images.append(resized)

where "img" are all the imageas read with "imread_collection" of skimage.io

Thanks in advance for your help.

EDIT1: the images resized with opencv have not been processed with the preprocessing function

Answer 1

I solved the problem following the advices in the comments of this discussion. I paste here my code:

dizionario = dict({'1': 0,
 '10': 1,
 '11': 2,
 '12': 3,
 '13': 4,
 '14': 5,
 '15': 6,
 '16': 7,
 '17': 8,
 '18': 9,
 '19': 10,
 '2': 11,
 '20': 12,
 '21': 13,
 '22': 14,
 '23': 15,
 '24': 16,
 '25': 17,
 '26': 18,
 '27': 19,
 '28': 20,
 '29': 21,
 '3': 22,
 '4': 23,
 '5': 24,
 '6': 25,
 '7': 26,
 '8': 27,
 '9': 28})

as you can see I created a dictionary to map classes label to index . I used the output of final_train_generator.class_indices to create this dictionary. After predict_generator I created the csv of prediction with following code:

predicted_class_indices = np.argmax(predictions, axis = 1)
final_predictions = []
for element in predicted_class_indices:
  final_predictions.append(list(dizionario.keys())[list(dizionario.values()).index(element)])
df = pd.DataFrame()
df['class'] = final_predictions
df['imnames'] = imNames
df.to_csv('predictions_xception_all_data_bon.csv', sep=',')

I paste here also the code that I changed following the discussion link in the comments:

test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)
test_generator = test_datagen.flow_from_directory( 
    directory="./TEST/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=20, 
    shuffle = False,
    class_mode = "categorical", 
    )
test_generator.reset()
imNames = test_generator.filenames
predictions = model_xcpetion.predict_generator(test_generator, steps=len(test_generator), verbose = 1 )

Answer 2

This behavior is because flow from directory assigns class labels lexicographically. So, all the folder names are treated not as int but string.

This is why you see folder '1' has class label 0. Folder '10' has class label 1 rather than Folder '2'.