What are the risks if the test data is significantly different from the training data?
Is the most significant problem related to both?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
The main risk is underfitting, a model trained on a significantly different dataset will poorly predict the test set
In order for the predictions to be as accurate as possible, the training data should be as representative as possible of the test data. The training data is never going to be exactly accurate, but should be as close as possible. Usually the best way to achieve this is by using a larger training set if possible or using random sampling if not.
If there is a significant difference then the biggest risk is that the model will under fit the test data and will give you inaccurate predictions.
You could also try splitting the training data into training and validation sets to see how the model works on that before applying it to your model and see how much of a difference there is.
Your model will be way off in terms of prediction accuracy(Underfitting the test dataset ), but that's not an issue because you can either collect more data and fine-tune your model as you miss predictions from unseen and far from what you trained on so you can cover all sorts of inputs in the long run. Or if your test inputs are always unpredictable, and your classes are imbalanced ( cancer detection example ) , train only on the dominant class examples, and label any input that is different from what you trained on as the dominated class.