Machine learning learn to work well on future data distribution?

Question

This is based on my limited machine learning scope and experience, so correct me if I'm wrong. Many of the currently used machine learning models (SVMs, boosted trees, DNNs) work under the assumption that the training, validation and test data sets share the same distribution. They can work to some extent if the distributions differ but not by a lot. Here "can work" means that they work sub-optimally (i.e. can work better if the distributions are the same), not that their theory behind is supposed to deal w/ the distribution difference and can handle them like "nailing it".

Hence my question: is there work on predicting based on the assumption that the data sets are actually moving through a series of distribution changes? A crazy thought would be to observe the distribution difference between training and validation sets, and assume that the same diff will exist between validation and test sets and learn to predict well on test set. This will work great on time series where the nature of the data might change over time.

Answer 1

It has been studied under various names likes Domain Adaptation, Sample Selection Bias, Co-variate Shift.

Please go trough this survey paper on Transfer learning. It covers all the possible combination like

1)same distribution for train and test data

2) Gradual change between train and test distribution

3) Different but related distribution for train and test

It'll also give you all the necessary resources required to study further on this topic.

Answer 2

A major assumption in many machine learning and data mining algorithms is that the training and future data must be in the same feature space and have the same distribution. This assumption is weak and in many cases may not hold. For example, imagine that we have a classification task in one domain of interest, but we only have sufficient training data in another domain of interest, where the latter data may be in a different feature space or follow a different data distribution.

You can find a lot of methods in the literature. Some of them are trying to tackle the difference in the marginal distributions p(X) of the train and test set, while others are trying to tackle the conditional distributions p(y|x). Some methods are taking into account both differences among the train and test set distributions. Almost all methods are trying to bring the two distributions closer, using weights on the train instances (importance sampling) so that the new weighted train set will be close to the test set. Other methods transform the feature spaces so that the distributions the train and test sets will be close enough.

For more information about this topic you can check the survey

Pan, Sinno Jialin, and Qiang Yang. "A survey on transfer learning." IEEE Transactions on knowledge and data engineering 22, no. 10 (2010): 1345-1359.