Keras single sample prediction returns different values

Question

I'm using a pre-trained ResNet50 model in keras and am trying to see predictions for single samples

The issue is that model.predict() is returning different values, depending on when I input a single sample vs the same sample within a larger array.

My model outputs a 1x6 vector of probabilities

I run on a single sample:

Case A: model.predict(X_test[0:1])

and within an array of multiple samples:

Case B: model.predict(X_test[0:2])

The prediction for X_test[0] is

Case A: [9.999e-01, 4.228e-06, 8.278e-05, 1.121e-06, 1.061e-06, 2.958e-05]

Case B: [1.000e+00, 7.702e-13, 2.969e-13, 6.343e-11, 7.477e-14, 1.304e-10]

Why are the predictions different? (predictions here being the probabilities themselves, not argmax)

As requested, the main code is below:

from keras.models import load_model
from my_utils import load_dataset
import numpy as np

X_train, Y_train, X_test, Y_test, classes = load_dataset()

model = load_model('model.h5') # ResNet50 with pre-trained weights

model.predict(X_test[0:1]) # Case A

model.predict(X_test[0:2]) # Case B

The model architecture of ResNet50 can be found here; the one difference with the implementation I'm using is that I'm using 6 classes not 1000

Thanks!

Answer 1

I do not how to fix this in a correct manner, but it seems the reason is because the model was trained on batches rather than individual examples. My model is also based on ResNet50 (a part of it) and I have a function that prepares datasets like this:

def to_ds(xy, batch_size=1):
    return tf.data.Dataset.from_tensor_slices(xy).batch(batch_size)

My observations: if I train my model on batch size 32 I have very high validation accuracy on the very first epochs (like 98-99). However, when I check my model on test dataset I see that the accuracy is much lower (75-85).

In order to understand what happens I had to write two versions of validation:

def ensure(x, y):
    # if I remove batch_size=1 these two functions give me different result
    predicted = model.predict(x, batch_size=1)
    diff = np.round(abs(predicted-y))
    print(np.sum(diff))
    print((x.shape[0] - np.sum(diff)) / x.shape[0])
    print(np.where(np.any(diff > 0, axis=1)))

def ensure2(x, y, limit=None):
    count = 0
    wrong_count = 0
    wrong_indexes = []
    for i in range(x.shape[0]):
        x_i = x[i]
        y_i = y[i]
        rez = model.predict(np.array([x_i]))[0, 0]

        rez_text = "ok" if abs(rez - y_i) < 0.5 else "wrong"

        if rez_text != "ok":
            show_as_image(x_i)
            print(rez, abs(rez - y_i), rez_text)
            wrong_count += 1
            wrong_indexes.append(i)
        
        count += 1
        if limit is not None and count > limit:
            break
    
    print(f'accuracy={(count - wrong_count)/count}')
    print(wrong_indexes)

I have googled this a bit and my understanding for now is that BatchNormalization layer somehow normalizes the values across all the samples in a batch, not the individual slices of data belonging to one sample. This is quite sad and not something that I need (in my case I want my model to be as accurate as possible on a single image passed, not 32 of them). So for now I am just switching to using batch_size=1 for my experiments.