1
$\begingroup$

Suppose I have want to predict how likely people in both US and Canada will buy product A in a month. Basically, it's a binary classification problem.

Assume I have 2 millions rows of historical data with people who has bought/not bought the product (label). One millions rows data are from US, the other are from Canada.

I can build a machine learning models in following two ways:

  1. Build a model with 2 millions rows of data. One model with all the data.
  2. Model localization: Build a model with US data. Build another model with Canada data.

Intuitively, it seems to me option 1 will always performs better than 2. Because there will be more data in one model. But in practice, option 2 seems always performs better than option 1.

My question is when should I use option 2 instead of option 1? How to make the decision?

$\endgroup$
  • $\begingroup$ Performs better on what evaluation criterion? What does your test data look like in terms of being from US or Canada? $\endgroup$ – Himanshu Rai Oct 11 '17 at 5:22
  • $\begingroup$ AUC. Assume I have one more month data as test/validation from both US and Canada. $\endgroup$ – nkhuyu Oct 11 '17 at 5:25
  • $\begingroup$ I would suggest KFold cross-validation before you can be certain of which model is better $\endgroup$ – Himanshu Rai Oct 11 '17 at 7:05
  • $\begingroup$ A third option is to include 'country' as a feature and let the model sort it out. $\endgroup$ – HEITZ Oct 12 '17 at 23:31
  • $\begingroup$ @HEITZ, That is what I mean for my option 1. So the question is how to make the decision to choose which one? Thanks. $\endgroup$ – nkhuyu Oct 12 '17 at 23:35
1
$\begingroup$

When and how to choose 1 vs 2? Data exploration, understanding the problem, and testing the models. Make sure you use metrics, a cut-off level and test data that is appropriate for your business problem.

Take for example, if there is a fundamental difference between US and CA customers. Then allowing the model to focus on the signal of each country, without dealing with the noise of the other country, may produce a better model for each. By doing data exploration (plot purchases by country over time and some other important features) you might convince yourself that it is worth the effort to test multiple models.

This is the art vs science of modeling.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.