So I have about 3000 images with 6 classes and this is what I did:

1 - split into training set and test set prior to anything with 20% test size

2 - performed data augmentation on the under represented classes in the training set and ended up with 2700 training and 640 test

3 - did feature extraction techniques (haralick, dominant color, avg color, hist, etc) on both sets

4 - did normalization of features using standard scaler (fit_transform on training and after just transform on test)

5 - did a gridsearch with 5 fold cv to find best params just in the training set and got 91% accuracy average

6 - used the best estimator to predict on the test set and got 94% accuracy

7 - pickled the model and scaler and then uploaded on a new file

8 - create a predict function with all the transformations and then feed it a random image from the data set, in theory this is not new data so it should give the same results yet it fails miserably every time

what am I doing wrong? I don't think its overfitting otherwise my test accuracy would fail I presume it's something to do with the scaler?

  • $\begingroup$ Did you also pickled your normalisation infos so you do exactly the same one on your new image than in your test ? $\endgroup$ – BeamsAdept Sep 30 '20 at 11:55
  • $\begingroup$ I used the same functions I created for the train transformations, for example I have a py doc with all transformation function: resize, split, etc. and imported those functions into the prediction pipeline $\endgroup$ – Marco Ramos Sep 30 '20 at 12:39
  • 1
    $\begingroup$ There can be a mistake here. Applying the same function can cause issue : imagine applying min/max normalization into only 1 data, then the min and max would be this exact value, so it would take 0.5 value. In fact you have to give the training known min and max to make the normalization with training known data. Same with One Hot Encoding for example, if you One Hot only one individual with Sex=Male, then it'll create Sex_Male = 1 and not create Sex_Female = 0 also used by model. $\endgroup$ – BeamsAdept Sep 30 '20 at 12:56
  • $\begingroup$ Can you share your code? Sometimes what you think you did is not what the code actually did. $\endgroup$ – Brian Spiering Nov 21 '20 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.