I created an XG Boost model to predict churn using a dataset of customers who were sold during 2018. The accuracy of the model is 89%. Does it make more sense to re-pull the 2018 dataset, where more customers would have churned, or to test against the 2019 data set?

Below is the model set up:

xgboost model

  • $\begingroup$ Can you describe your data a bit more here? From my understanding for churn rate problems, you are supposed to be predicting a binary outcome 1/0 over some fixed time period (ex: churn or not churn over four months), or you can approach the problem as a survival analysis, where a "death" is a cancellation? Why are we specifying objective = "reg:linear" (regression problem, continuous variable) but using accuracy (classification, discrete variable) as our evaluation metric? $\endgroup$ – aranglol Jul 12 '19 at 2:53
  • $\begingroup$ @aranglol - Sure thing.The data is about 160 variables (categoricals are one-hot encoded) with Churn being the binary 1/0 prediction (1 = churn, 0=no churn). Regarding the "reg:linear", are you saying I should specify the objective ='reg:linear" to something else such as "squarederror" or "binary:logistic"? (I did not realize "reg:linear" is the default setting.) $\endgroup$ – flyeaglesfly Jul 12 '19 at 14:39
  • $\begingroup$ You should specify the objective = 'binary:logistic' because your target variable is binary discrete. "reg:linear" = 'reg: squarederror' is typically used for regression problems (when your target variable is fully continuous, like age, weight, stock price, etc.). The choice actually is not completely incorrect (because the mean squared error ends up being minimized, which for classification problems, ends up being the Brier score) but indeed, there are good statistical properties of the loss function for "binary:logistic" related to maximum likelihood estimation and the likelihood principle. $\endgroup$ – aranglol Jul 12 '19 at 20:36
  • $\begingroup$ Thus, I would recommend changing 'reg:linear' to 'binary:logistic' unless you have good reasons to be minimizing a loss function that is perhaps less suitable with a two class classification problem that assumes a bernoulli/binomial distribution. By specifying 'binary:logistic' you directly maximize the likelihood function of a binomial distribution which for basically all binary classification problems is a sound choice. $\endgroup$ – aranglol Jul 12 '19 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.