1
$\begingroup$

I have read in literature that in some cases the training set is not representative for a real-world dataset. However, I cannot seem to find a proper term describing this phenomenon; what is the proper term to address this problem?

Edit:

So far I have settled for the term domain adaptation, shortly described as a field in machine learning which aims to learn from a certain data distribution in order to predict data coming from a different (but related) target distribution.

$\endgroup$
  • 1
    $\begingroup$ I first thought of transfer learning, which is referenced on the domain adaptation wiki, but it sounds like domain adaptation may be more appropriate for what you are describing. $\endgroup$ – foutandabout Dec 25 '16 at 14:43
2
$\begingroup$

You may be looking for sampling bias. Also the other case (where training set does in fact represent the real world data-set well) is generally known as representative sample.

Hope this helps.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I've come across that term, however, it is (in my case) not the term that I was looking for. The cause of the discrepancy is more likely to be characterized by the term domain adaptation. But I am looking for a more general term or discipline that addresses the issue that training sets can not be used for their real world test sets. $\endgroup$ – Archie Sep 1 '16 at 8:07
0
$\begingroup$

Extrapolation? Happens a lot when your data distributions change over time, so a system that is well modeled in the training set wont know how to deal with values that are not in a similar range. More of a general term, so it might be what you're looking for.

It also has different effects depending on the technique you use. Something like random forests is not very good at extrapolation, where others like logistic regression can still perform OK.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Overfitting?

This happens when you make a model too specific to the training set, so that it performs very well on that particular training data, but then it is not able to generalize to other data ("real world data") and so performs poorly in reality.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Not what I am looking for, as this is dataset dependent, and the possible error might be due to a change in the distribution of the real-world dataset. Therefore I have settled for the term domain adaptation. $\endgroup$ – Archie Nov 3 '16 at 9:50
0
$\begingroup$

The case that you are describing is referred in the literature as sample selection bias [1]. This case is a part of the area of Transfer Learning/Domain Adaptation. The training set does not represent the real world data-set well, which means that there is a difference between the distributions of the training and test sets. Another term from the Domain Adaptation area that is referred to the same problem is the Covariate Shift.

  1. B. Zadrozny, “Learning and Evaluating Classifiers under Sample Selection Bias,” Proc. 21st Int’l Conf. Machine Learning, July 2004.
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.