How to improve machine learning model using 2+ datasets

Question

I am building a supervised machine learning model which (for example) predicts heart failure (yes/no). I have two datasets from 2 different labs A and B, which both have decent distribution, aka it's not like A has way more young people than B, but somehow A has much lower rate of heart failure.

Having created a separate model for each, they each achieve around 90% or higher accuracy, but my aim is to make one model which can use a concatenated dataset containing information from A and B. Currently this combined model gets 75% accuracy for samples from B and 90% for A.

What I have tried:

• sample in different ways / create 'even' datasets
• add feature indicating which lab
• normalise numerical features
• create categories for numerical features

I'm aware the overall results would of course not become as good as the individual models, but how can I make the platforms at least get more similar results? I appreciate any advice / methods I could try out to tackle this type of problem!!

Answer 1

You may wish to combine the two models using an approach called Ensemble Methods.

However, in your case I'm surprised that using the combined data in a single model is so much worse if as you say the distribution of the features is similar.

So two potential outcomes:

1. if the distributions of the data are the same, the it could be the individual models are overfitted and given the larger data set this is exposed.

2. if the distributions are different then each model is being tuned for specific features and hence a single general model is hard to achieve.

To get a better feel of what to expect, I would suggest doing some distribution plots for each feature to see if they are as you say similar.

Another simple test is to take Model A and use it to predict everything in Dataset B and vice-versa. If the actual accuracy is around 70% then everything is working as expected.