I am curious to know how training data should be constructed so that it scales to examples that are not a part of the training data. For example, the problem that I am facing right now is in the application of identifying or distinguishing the frequency response of time series that are generated from different distributions. So I constructed $p$ number of examples each from Gaussian, Uniform, Poisson and a kind or colored noise say pink. The White noise examples (Gaussian, Uniform and Poisson) are labelled as 1 and colored noise as 0. Using Neural Network the classification works fine. Now I wanted to do sensitivity analysis by checking if the trained network can classify white noise from another distribution and also colored noise say red. Both the tests failed. NN failed to classify them. But, as soon as I included the red and the new kind of white noise in the training data and tested on a different trail (time series), the NN could classify it.

QUESTION: This behavior makes me wonder if machine learning algorithm sare incapable of distinguishing examples from different systems eventhough the examples in testing have similar properties as those used in training. In this case, eventhough white noise appears similar but since they are generated from different distribution or say systems the training data must include examples from all the generating mechanism or systems, otherwise in testing the ML model fails to recognize it. Is this the usual behavior?


One of the basic assumptions governing Machine Learning is that samples from the training set must follow the same underlying distribution as samples from the test set (and so must any other sample you want fed to your model)!

This is why, usually, we randomly partition the same dataset into training and test sets.

This is actually one of the main reasons ML models underperform in some real world applications. You might have trained your model in a specific dataset, but overtime the data slightly changed its characteristics and new data differ from the old ones that were used to train the deployed model. In this case you need to retrain your model on the new data.

  • $\begingroup$ oh I see....can you provide any links on how to check for the distribution being alike or not ....how to ensure that the new unseen data set follows the same distribution $\endgroup$
    – Sm1
    Aug 26 '19 at 16:07
  • $\begingroup$ Unfortunately, this isn't as easy as it seems... Generally speaking you want to sample the training and the test sets from the same origin. Unfortunately I don't know of any easy way of ensuring this. If they aren't though it falls out of the scope of supervised learning. Applying extracted knowledge from one source to another (somewhat unrelated) falls under the scope of transfer learning. $\endgroup$
    – Jerome
    Sep 2 '19 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.