2
$\begingroup$

I would like to ask about the theoretical approach of using Logistic Regression for customer data and more specifically Churn Prediction (in BigQuery and Python).

I have my customer data for an online shop and I would like to predict if the customer will churn based on some characteristics. I have created my dataset and the Churn label (based on the hypothesis that if the customer hasn't bought something in the last year then it is assumed that the customer is churned since we are dealing with a non-contractual setting).

I am using 3 years of data (2019-2021), which includes ~3M customers and 43 features, and as I said, a customer is considered to be churned if the customer didn't place an order in 2021.

  1. I checked the distribution of my label which is ~balanced.
  2. I checked for some Logistic Regression assumptions such as multicollinearity, outlier influence etc.
  3. I split the data into 80% training data, 10% evaluation data, 10% prediction data.
  4. I checked the model's performance by looking at the classification metrics (Accuracy, Recall etc.)

My question would be:

We have the predictions of the 10% of the data (i.e. the probabilities that a customer will churn). Could we have the probabilities for all the other customers that belong in the training dataset and in the evaluation dataset?

In other words, what would be the next steps after we have trained and have checked that we could use the model, if your final goal would be to have in the end the probabilities of your customers to churn or to not churn?

Thank you in advance for your help!

$\endgroup$

2 Answers 2

0
$\begingroup$

You have method of your trained that model that will return you the predicted probability:

model.predict_proba(X)

Check the reference for more information and examples.

$\endgroup$
4
  • $\begingroup$ Thanks @Alex for your answer, but it's not what I am asking :). As I also wrote in my question, I already have the probabilities for the testing set (i.e. prediction of the class for the 10% testing data). $\endgroup$
    – Ledian K.
    Commented Apr 22, 2022 at 8:56
  • $\begingroup$ So which is your question exactly? I mean, if you have already trained the model, is time to use it with new data. You can do something like: model.predict(real_incoming_data). Provide which will be your desired output. $\endgroup$ Commented Apr 22, 2022 at 9:06
  • $\begingroup$ Using different numbers just to make it easier. I have 1M customers in total that I would like in the end to predict if they will churn or not. I use 800K for training, 100K for evaluation and I see that my model is working fine. Then, I apply my model to the remaining 100K as prediction in new data. My question is how do we get the predictions for the 800K (training) and 100K (evaluation). I am thinking that if I use the model to predict the training data, wouldn't that lead to overfitting? $\endgroup$
    – Ledian K.
    Commented Apr 22, 2022 at 9:19
  • $\begingroup$ Every model is trained on data (of course). If you evaluated your X_train data, probably you will be overfitting. Another solution, is created another model with different data. But this is a situation that you will find in every model you create. It is assumed that training data is not needed to be evaluated, that's the main idea. Evaluating it, it makes no sense at all. $\endgroup$ Commented Apr 22, 2022 at 12:02
0
$\begingroup$

I broadly agree with the comment on the other answer:

It is assumed that training data is not needed to be evaluated, that's the main idea. Evaluating it, it makes no sense at all.

Your supervised learning method had to have ground truth labels to learn from, and so your model's predictions on the training dataset won't be any better than those labels.

But, what if your labels aren't necessarily the whole truth? As in your case, you've set some definition/proxy for "churn", but perhaps the model has learned something a bit broader than that definition just because the independent variables are better at finding that than your proxy. It's a stretch, and you can't expect it, but perhaps looking at notable "failures" of your model to align with the target (so e.g. low-probability-of-churn according to your model but didn't purchase in 6 months) can be insightful. Or in other settings, maybe the data labels are just mistaken some percentage of the time.

To tell, you're going to have to do some manual legwork: dig into those cases, and see if you believe your label or your model.

You'll want to avoid models that overfit easily for this, and/or consider k-fold predictions so that the predictions for each row are always by a model not trained on that row.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.