Splitting data when combining multiple datasets

Question

I have 13 small datasets from 12 different countries. All datasets have the same outcome and features, though have a different number of observations (ranging from ~50 to ~800). I would like to combine these datasets in a ML model.

Based on the answer to this question (Is it advisable to combine two datasets?), I can simply include a feature identifying the source of the data to control for potential bias.

Assuming this is true and extending on that question, what is an appropriate way to split the data into training/testing sets (ie, should I sample such that each dataset has the same representative proportion)? With 13 datasets, would it be advisable to leave out 2-3 datasets entirely from model development for external validation, and if so what would the decision-making process be for that?

Answer 1

First, when doing exploratory data analysis, I'd compare the distributions of each feature across each dataset to see how they differ across locations. Boxplots and barplots are great for this.

When combining all the datasets, you can split into training and testing so that not only are there equal proportions of your target in the train and test datasets, but so that there are equal proportions of each dataset location in your train and test datasets (e.g. 10% of total dataset is country 1 in train and 10% of total dataset is country 1 in test).

The decision-making for leaving 2-3 datasets out as holdout should depend on the size of your datasets and the question you're trying to answer. If doing a 11-2 (training-testing) split or 10-3 split or 12-1 split answers your question, do it. You can try each combination and see which dataset combo may have more generalizability or less.