2
$\begingroup$

My model is based on Shallow Net.
When I am training my model, the results are:

loss: 1.1398 - accuracy: 0.6093 - val_loss: 1.2309 - val_accuracy: 0.5657

Then I downloaded 20 images (2 for each class) from the net to check the performance.

enter image description here

Labels corresponding to this dataset should be:0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9].
But my model's prediction is: [0,0,0,1,0,1,5,0,0,5,2,2,0,0,5,2,0,0,1,9].

The accuracy is: 0.2 which is quite low as compared to 0.5657.

My code to load these datasets:

for file in os.listdir("C:/Users/....."):
    img_arr=cv2.imread(os.path.join(os.getcwd(),"Dataset",file))
    img_arr=cv2.resize(img_arr,(32,32))/255
    img_arrs.append(img_arr)

img_arrs=np.array(img_arrs)
img_arrs=img_arrs.reshape(20,32,32,3)
model=load_model("weights.hdf5")
pred=model.predict(img_arrs).argmax(axis=1)

What could be the reason behind this? Can someone give me an insight?

Edit:(Added training code)

(x_train,y_train),(x_test,y_test)=cifar10.load_data()
x_train=x_train.astype(float)/255
x_test=x_test.astype(float)/255
lb=LabelBinarizer()
y_train=lb.fit_transform(y_train)
y_test=lb.transform(y_test)

labelNames = ["airplane", "automobile", "bird", "cat", "deer","dog", "frog", "horse", "ship", "truck"]

model=ShallowNet.ShallowNet.build(width=32, height=32, depth=3, classes=10)
sgd=SGD(0.001)
model.compile(optimizer=sgd,loss="categorical_crossentropy",metrics=["accuracy"])
H=model.fit(x_train,y_train,validation_data=(x_test,y_test),batch_size=32,epochs=50,verbose=1)
$\endgroup$
2
  • $\begingroup$ To me, your question seems to me a bit "too open" since there are MANY things that can cause this issue. Could you try to explain what you did in brief words, so we can understand your level of knowledge, and try to understand if that's a basic error on the process, or more a complex tricky one ? $\endgroup$ – BeamsAdept Nov 4 '20 at 15:20
  • $\begingroup$ Thanks @BeamsAdept for your time, I added my training code. $\endgroup$ – Shiv Nov 4 '20 at 17:42
2
$\begingroup$

Here are some possibilities that come to mind:

  • The ordering of the dimensions. Depending on the network you are training with and the original training data, you may need to transpose some of the dimensions and/or reshape them differently. The original cifar 10 data is a little strange in that the color-dimension precedes the height/width dimensions. It may help to visualize both an image from the training data and the internet data side by side with the same function and ensure they are displaying the same way as a test of this.
  • Was any pre-processing performed on the training images, for example rescaling of values? If so, the same pre-processing should be performed on these images. Even if pre-processing was not explicitly performed image data can be stored in a variety of different ways. I would check the range of values on your training/validation images and compare against the images you are getting from the net.
  • Something funky is possibly happening when these images are resized. I would suggest visualizing the above examples after they have been resized and confirming that they are reasonable.
  • Another possibility is that something unexpected is happening when loading the model - it may seem redundant given the validation results but I would nonetheless take a set of 20 or so images from the original validation set and check the predictions after retrieving the model with the above load_model function.
$\endgroup$
6
  • 1
    $\begingroup$ To complete, another thing that comes to my mind is the possibility of bad evaluation of your model. Make sure you evaluated your model on a test sample, with images you didn't use to train your model. If you evaluate your model on the same sample you train on, the model will know these images well, since it was build using them, so the performance will be over estimated $\endgroup$ – BeamsAdept Nov 4 '20 at 15:18
  • 1
    $\begingroup$ Thanks @James, I visualized every image after resizing they were very small but were distinguishable. $\endgroup$ – Shiv Nov 4 '20 at 17:50
  • $\begingroup$ I have taken less number of images (2 per class) for each classes, can this be the reason my accuracy is affected? $\endgroup$ – Shiv Nov 4 '20 at 17:51
  • 2
    $\begingroup$ Hmm, a larger number of classes would give you a more accurate picture of the true accuracy but I don't think this is the primary problem. A couple of other things to consider though. os.listdir does not necessarily return the files in alphabetical order so I would be printing file names to confirm their order. Another thing is, are you certain the LabelBinarizer is necessary here. I think the labels you get from the load_data function are already integer encoded no? By applying the LabelBinarizer you may be mixing up these codes to be different to what you would expect. $\endgroup$ – James Nov 5 '20 at 1:10
  • $\begingroup$ Images listed in the directory are in alphabetical order, and listdir returns the files as they are listed in the directory. Also, Y from load_data in this case, needs to be encoded. I double checked both the above points. $\endgroup$ – Shiv Nov 5 '20 at 13:27
0
$\begingroup$

I think ~50% accuracy might not be so good to match it to new images on the basis of accuracy only.
Similarly 2 images per class is also a bit small to check a model unless a model is built on millions of images which has seen almost every type of variances across pixels

What you may try -

Check the Loss of Individual classes on training
Try to match the Loss of these images i.e. 2 per class
Do the same thing for Accuracy i.e. per class accuracy
Try to check the activation of 2nd last layer i.e. layer prior to Softmax
Try to gather 80-90 images and repeat if your Model score ~80%

I meant you should try to look inside the Model instead of a Black-box approach

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.