1
$\begingroup$

A colleague and I are working on a churn model and reached an impasse:

Our data set is for a global product. We've been asked to look at the US market only.

When we subset the data to the US only, the classifier evaluation metrics are lower than when we use the total global data set.

My colleague wants to use the global data set because the output metrics are higher. I consider this the wrong thing to do, we should limit the data to the US market only.

My thinking: Only use the data set that best represents the situation you are looking to explore. That is, we should only be using the data set filtered to the US market here.

As we're dealing with human shopper behavior here, there could be lots of localised factors that change from market from market - culture, salary, shopper behavior, localised competitors.

Is the approach to use the filtered data set correct? Are there papers, similar that talk to this point? Is there a useful term to search on Google for?

$\endgroup$
14
  • $\begingroup$ What is the kind of data (e.g., tabular, time series)? What classifier are you using? What is the difference in magnitude between the two datasets (global vs USA-only)? It is a multi-class classification problem? If so, perhaps the classifier on the global dataset is better on other classes, and this increases the metrics, whereas with a subset of it (USA-only) the model gets confused. $\endgroup$
    – Eduard
    Commented Aug 23, 2022 at 11:09
  • $\begingroup$ @0xedu The data is tabular, tree-based classification to predict the probability of someone churning. For magnitude, what are you thinking of, size of data set? $\endgroup$ Commented Aug 23, 2022 at 12:25
  • $\begingroup$ Just to understand better your scenario. You are predicting a class or a number? As for the magnitude, yes, just to have an insight into the proportions of the smaller dataset compared to the bigger one; that is, I'm not interested in how many bytes they occupy in memory. $\endgroup$
    – Eduard
    Commented Aug 23, 2022 at 13:14
  • $\begingroup$ Are your evaluation metrics measured for just the US? $\endgroup$
    – Ben Reiniger
    Commented Aug 23, 2022 at 13:20
  • 1
    $\begingroup$ I'm suggesting what Dave's answer says: try two models (as you have), one trained on US and one on global, but (as you haven't) to score each model on a US test set. See how the two perform and then choose which to take. I agree with Dave's response that both approaches have theoretical merit (data volume or data representativeness), and the best way to pick is to see how they both perform on the task you wish to perform. But scoring models on different test sets is comparing apples and oranges. $\endgroup$
    – Ben Reiniger
    Commented Aug 25, 2022 at 14:01

1 Answer 1

2
$\begingroup$

You could consider it a hyperparameter and tube it to the best value.

As you point out, there are multiple possibilities. Your stance of using only the most representative data has merit; the stance of using all available data has merit, since more data results in tighter estimates, and nothing says that Americans have to be so unique.

Therefore, go figure out which approach gives the best results.

Early evidence suggests that using more data results in better performance.

$\endgroup$
2
  • $\begingroup$ Thanks @Dave. Are you suggesting that the number of rows (or even countries) becomes a hyperparameter to then pass into grid search? [Based on the countries comment, I wonder what happens if more countries "similar" to the USA were included and then countries dissimilar to the USA were excluded] $\endgroup$ Commented Aug 23, 2022 at 12:30
  • $\begingroup$ I wasn’t thinking of being that formal about it, but sure, you could expand the set of possibilities and then search over the whole space like you might for other hyperparameters. $\endgroup$
    – Dave
    Commented Aug 23, 2022 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.