Dividing data between test, learn and predict

Question

I was posting on stats.stackexchange but perhaps I should be posting here.

Context. Subscription business that charges users a monthly fee for access to the service. Management would like to predict "churn" - subscriptions who are likely to cancel. Management would like to create an email sequence in attempt to prevent high risk accounts from churning, perhaps with a discount code of some sort. So I need to identify those accounts at risk of leaving us.

I have a dataset with say 50k records. Each line item is an account number along with some variables. One of the variables is "Churned" with a value of "yes" (they cancelled) or "No" (they are active).

The dataset I have is all data since the beginning of time for the business. About 20k records are active paying customers and about 30k are those who used to be paying customers but who have since cancelled.

My task is to build a model to predict which of the 20k active customers are currently likely to churn.

Here is where I have tied my brain in a knott. I need to run the model (Predict) on the 20k records of active customers.

How do I split my data between training, test and predict?

Does predict data have to be exclusive of train and test data?

Can I split the entire dataset of 50k into 0.8 train and 0.2 test, build a model and then predict on the 20k active accounts? That would imply I'm training and testing on data that I'm also going to predict on. Seems "wrong". Is it?

Answer 1

Supervised Learning:

Do you have a saved time history of the data? For a supervised learning set you need some churned="No" cases and some churned="Yes" cases, but it sounds like you only have churned="Yes" and the unknown cases e.g. current customers who may or may not churn. With some time history you can go back in time and definitively label the current customers as churn="No".

Then it is very easy to split up the data. And no, you probably don't want to predict on any data that you trained on since you can only train on it if you already know the solution so it will be a waste of time and throw off any metrics you might use to assess accuracy (precision/recall/F1) in the future.

Unsupervised Learning:

If you don't have saved time history of the data then this is an unsupervised learning set for which you have churned="yes" and churned="maybe". You could then employ anomaly or outlier detection on this set.

novelty detection: The training data is not polluted by outliers, and we are interested in detecting anomalies in new observations.

outlier detection: The training data contains outliers, and we need to fit the central mode of the training data, ignoring the deviant observations.

You can do either one but novelty is more powerful. This is kind of a flip around as the novelty here is Churned="No" since all of your data is the confirmed Churn="Yes" cases.

Hope this helps!

Answer 2

The docs from scikit-learn can help with best practices for splitting your data for cross-validation (training, testing, validating): http://scikit-learn.org/stable/modules/cross_validation.html

In your case, all of your accounts are labeled as "Churned" with values either "Yes" or "No". If you make a model that predicts Churned=Yes/No, then you're answering the question "Is a given account currently active?", which I don't think is the question you want to answer.

If I understand, I think you want to ask an actionable question about the Churned=No accounts:

• "What is the probability that a Churned=No account will become a Churned=Yes within the next 30 days?"
  • Requires access to time-series data.
  • Example approach:
    • Use the 30k Churned=Yes accounts to create a supervised model (since you know when and how these accounts went from Churned=No to Churned=Yes) and apply it to the 20k Churned=No accounts.
    • The predicted quantity is the probability of a Churned=No account becoming a Churned=Yes account within the next 30 days.
• "How 'close' is a Churned=No account to being a Churned=Yes account?"
  • Does not require access to time-series data.
  • Example approach:
    • Again use the 30k Churned=Yes accounts to create a supervised model (may require multiple models for different clusters of similar accounts) and apply it to the 20k Churned=No accounts.
    • The predicted quantity is a metric (i.e. a risk score) that measures the proximity in parameter space of a Churned=No account to the Churned=Yes accounts.