How to normalize data from multiple sources?

Question

I am trying to model an individuals' purchasing behavior using different data sources (ex: Zalando, Otto, etc.,). When I combine data sources, I see that the data across these channels is very different.

For example, 5% heavily shop using a particular channel, but do not utilize other channels. When I try to model anything from this information, it performs poorly because overall, its a sparse dataset, but a % of each column should be a very good predictor of a small subset of population.

My question is: How do I combine/normalize such a dataset wherein the data is super sparse?

Answer 1

Make sure that when you merge your datasets, the column information is as generic as possible, which means: When you merge A with B, make sure you name "Purchases at (any of the) online channel" not "Purchases at A's online channel" and "Purchases at B's online channel.

Merging datasets from completely different datasources is very difficult because they were not thought with the same basic ideas.

Answer 2

I don't think it would be possible to merge them and normalize the data to rebalance the classes.

Here is an example case of the problems that may arise.

You had 2 data sources, one small with mostly male and one larger with mostly female and you want to merge them together. If you know the population distribution, you could randomly sample out some examples. This becomes difficult if you have a lot of different variables and/or don't know their distributions.

What you could try is to find the size of different data sources (trafic, nb users, etc) and try to match this distribution in your final merged dataset (larger sites have bigger proportion of events in merged dataset, since they probably represent a larger part of the population)