1
$\begingroup$

Neural Net accuracy is high on test set but low on new real world image examples.

Looking for advice regarding what generally causes this scenario and how to fix it.

Sampling basis? Training/test set is not representative of real world data? Obtain more training/test data?

$\endgroup$
  • 1
    $\begingroup$ Probably your train and test set are not representative, but it is impossible to tell with no information. Can you tell the difference between train/test and real world data yourself? Can you show samples? How much data do you have? More relevant information will be helpful in getting you good answers. $\endgroup$ – Simon Larsson Apr 12 '19 at 7:08
  • $\begingroup$ Train/test/real world data are of the same format/quality. ~10,000 positive training examples and ~10,000 negative training examples for binary classification. $\endgroup$ – joshvarial Apr 13 '19 at 2:00
3
$\begingroup$

Many resources teach the process of splitting data into training, validation and test sets. This is what you want to do for "closed" datasets where it's not possible to get additional data.

This assumption of a closed dataset is often not true in the real world, where it may be feasible to collect more data. Statistically speaking, it is a lot more desirable to define a test set as a new data sample that was collected separately from your training data. This might be more representative of how the model will behave in production, but sometimes even this is not enough:

A few weeks back I built an image classifier for cars. I trained it using a mix of existing datasets and the results of a web scrape. Ultimately, it was deployed through an iOS app where it was supposed to do make predictions in real time. In this case, it was not enough to just create a test split or collect a new sample from the web. We needed to shoot our own images that were representative for the use case in order to make realistic assumptions about the app's performance.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.