I wonder if transfer learning can be used in tabular data similarly to how it's used in neural networks for image recognition. My idea would be to train a "general" model and then "localize" it using a specific dataset.

I have a problem akin to this one: I have a firm with customers who may churn every year. I operate in 3 countries, let's say Germany (45% share), US (45% share), and Holland (10% share). I have a separate churn model for each country, where local features, such as City are some of the most important features.

My Holland model is lacking and I would like to improve it using data from Germany and US, where I have a lot more data. The datasets share many features, such as Age, Products purchased, Socio-economic category, so there is information to be transferred from Germany and US to Holland. If it's relevant, I am using boosted trees (catboost) in my current model.

How would I effectively train a Holland model using info from Germany and US and is this worthwhile based on your experience?

Here are my ideas:

  1. Baseline: Training a large model with all 3 countries (including City variable) and using its predictions for Holland.

Issue: The model minimizes total error and Holland's small share (10%) causes the country-specific error to be large, possibly worse than for the individual Holland model. The training weight of Holland might be increased, but my intuition is the model is still not as effective as it could be, since it's using lots of splits for local variables such as City for Germany and US, which are useless for the Holland model.

  1. Training a 3-country model with "general" features only, i.e. those that are shared across all 3 countres (e.g. Products purchased).

2 a) Then outputting churn probability from the 3-country model and inputting that as a new feature in the localized Holland model, which also includes "local" features like City (Stacking.)

2 b) Taking the trained 3-country model from (2) with only general features and using its parameters as a warm start for a new localized Holland model. This is done for Neural networks, I am not sure if/how this can be done for boosted models and what to do if the features are not the same.

  1. Training a large 3-country model, but deleting values for "local" variables such as City for non-focus countries, i.e. making them NULL for Germany and US and keeping intact for Holland.

Issue: I kind of like this approach, the only downside is again low weight for Holland. Once we increase the weight (e.g. 70%), then the benefit of using extra data from Germany and US is diminished.

Final note: This paper (described in blog here) proposes a solution using Neural networks, in my understanding their approach would be to train a 3-country model and then fine-tuning the NN weights for the Holland-only model (and they have a trick about unifying different features), so close to approach 2b) described above. Has anyone tried it?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.