May the training sets of two predictive modeling cases (with an overlap in features) be combined?

Question

Say we have dataset D1 (columns A, B, C) and D2 (columns A, B, D) with target variable E. As both datasets are rather small, their respective predictive models do not perform really well. To improve predictive performance, may/should I apply the steps below?

• Remove the not-in-common columns (so column A, B and E remain in both datasets)
• Split each dataset in a train and a test set (cross-validation to evaluate, so no validation set)
• Train our predictive model on the combination of the training sets of D1 and D2
• Assess test set performance of D1 and D2 on their respective test sets (using 1 model rather than 2)

I assume this is allowed, but am not fully sure if there are any (bad) repercussions.

Answer 1

Well, there is a obvious trade-off of columns vs rows. Why don't you first check the importance of columns C and D in the individual models? The less relevant they are, the better idea it is to throw them away and comine the tables.

For example, you can try a random forest with sklearn and then check feature_importances_.