3
$\begingroup$

Suppose I have want to predict how likely people in both US and Canada will buy product A in a month. Basically, it's a binary classification problem.

Assume I have 2 millions rows of historical data with people who has bought/not bought the product (label). One millions rows data are from US, the other are from Canada.

I can build a machine learning models in following two ways:

  1. Build a model with 2 millions rows of data. One model with all the data.
  2. Model localization: Build a model with US data. Build another model with Canada data.

Intuitively, it seems to me option 1 will always performs better than 2. Because there will be more data in one model. But in practice, option 2 seems always performs better than option 1.

My question is when should I use option 2 instead of option 1? How to make the decision?

Improve this question
$\endgroup$
6
  • $\begingroup$ Performs better on what evaluation criterion? What does your test data look like in terms of being from US or Canada? $\endgroup$
    – Himanshu Rai
    Oct 11, 2017 at 5:22
  • $\begingroup$ AUC. Assume I have one more month data as test/validation from both US and Canada. $\endgroup$
    – nkhuyu
    Oct 11, 2017 at 5:25
  • $\begingroup$ I would suggest KFold cross-validation before you can be certain of which model is better $\endgroup$
    – Himanshu Rai
    Oct 11, 2017 at 7:05
  • $\begingroup$ A third option is to include 'country' as a feature and let the model sort it out. $\endgroup$
    – HEITZ
    Oct 12, 2017 at 23:31
  • $\begingroup$ @HEITZ, That is what I mean for my option 1. So the question is how to make the decision to choose which one? Thanks. $\endgroup$
    – nkhuyu
    Oct 12, 2017 at 23:35

1 Answer 1

Reset to default
2
$\begingroup$

When and how to choose 1 vs 2? Data exploration, understanding the problem, and testing the models. Make sure you use metrics, a cut-off level and test data that is appropriate for your business problem.

Take for example, if there is a fundamental difference between US and CA customers. Then allowing the model to focus on the signal of each country, without dealing with the noise of the other country, may produce a better model for each. By doing data exploration (plot purchases by country over time and some other important features) you might convince yourself that it is worth the effort to test multiple models.

This is the art vs science of modeling.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.