3
$\begingroup$

I'm using a pre-trained ResNet50 model in keras and am trying to see predictions for single samples

The issue is that model.predict() is returning different values, depending on when I input a single sample vs the same sample within a larger array.

My model outputs a 1x6 vector of probabilities

I run on a single sample:

Case A: model.predict(X_test[0:1])

and within an array of multiple samples:

Case B: model.predict(X_test[0:2])

The prediction for X_test[0] is

Case A: [9.999e-01, 4.228e-06, 8.278e-05, 1.121e-06, 1.061e-06, 2.958e-05]

Case B: [1.000e+00, 7.702e-13, 2.969e-13, 6.343e-11, 7.477e-14, 1.304e-10]

Why are the predictions different? (predictions here being the probabilities themselves, not argmax)

As requested, the main code is below:

from keras.models import load_model
from my_utils import load_dataset
import numpy as np

X_train, Y_train, X_test, Y_test, classes = load_dataset()

model = load_model('model.h5') # ResNet50 with pre-trained weights

model.predict(X_test[0:1]) # Case A

model.predict(X_test[0:2]) # Case B

The model architecture of ResNet50 can be found here; the one difference with the implementation I'm using is that I'm using 6 classes not 1000

Thanks!

$\endgroup$
5
  • $\begingroup$ Can we see the code you are using please? $\endgroup$
    – JahKnows
    Commented May 21, 2018 at 5:42
  • 1
    $\begingroup$ dear @Chaney you have not added the architecture of your model. If you have not coded yourself, add the output of model.summary() here. $\endgroup$ Commented May 21, 2018 at 6:07
  • $\begingroup$ Hi @Media, the model I'm using is ResNet50 (architecture can be found here: link, with minor difference that I'm using 6 classes at end, not 1000). The output of model.summary() is similar to what is in the link but can add if still helpful $\endgroup$
    – Chaney
    Commented May 21, 2018 at 6:29
  • $\begingroup$ maybe load_dataset() are randomly shuffling the data? $\endgroup$ Commented May 21, 2018 at 8:07
  • $\begingroup$ Hi @Chaney, Did you solve the issue? I have run into the same problem right now $\endgroup$
    – paradox
    Commented Nov 23, 2018 at 15:13

1 Answer 1

0
$\begingroup$

I do not how to fix this in a correct manner, but it seems the reason is because the model was trained on batches rather than individual examples. My model is also based on ResNet50 (a part of it) and I have a function that prepares datasets like this:

def to_ds(xy, batch_size=1):
    return tf.data.Dataset.from_tensor_slices(xy).batch(batch_size)

My observations: if I train my model on batch size 32 I have very high validation accuracy on the very first epochs (like 98-99). However, when I check my model on test dataset I see that the accuracy is much lower (75-85).

In order to understand what happens I had to write two versions of validation:

def ensure(x, y):
    # if I remove batch_size=1 these two functions give me different result
    predicted = model.predict(x, batch_size=1)
    diff = np.round(abs(predicted-y))
    print(np.sum(diff))
    print((x.shape[0] - np.sum(diff)) / x.shape[0])
    print(np.where(np.any(diff > 0, axis=1)))

def ensure2(x, y, limit=None):
    count = 0
    wrong_count = 0
    wrong_indexes = []
    for i in range(x.shape[0]):
        x_i = x[i]
        y_i = y[i]
        rez = model.predict(np.array([x_i]))[0, 0]

        rez_text = "ok" if abs(rez - y_i) < 0.5 else "wrong"

        if rez_text != "ok":
            show_as_image(x_i)
            print(rez, abs(rez - y_i), rez_text)
            wrong_count += 1
            wrong_indexes.append(i)
        
        count += 1
        if limit is not None and count > limit:
            break
    
    print(f'accuracy={(count - wrong_count)/count}')
    print(wrong_indexes)

I have googled this a bit and my understanding for now is that BatchNormalization layer somehow normalizes the values across all the samples in a batch, not the individual slices of data belonging to one sample. This is quite sad and not something that I need (in my case I want my model to be as accurate as possible on a single image passed, not 32 of them). So for now I am just switching to using batch_size=1 for my experiments.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.