1
$\begingroup$

Good Day, I am newer to data science so I am not confident in this. To set up the question I will describe my data and approach.

Data

I don't want to share specific data examples as I want to try and keep some anonymity. I have data for events between 2016 and 2019. Each event has ~14 features (categorical and numerical) as well as a binary label of success or failure. Each of these is run through different transformers (normalizing and one-hot encoding).

Approach

What I am interested in doing is knowing how likely events are to success (not necessarily predict if they will succeed).

I played around with train/test splits to find an algorithm that worked best. I ran a stratified k-fold cross-validation on the data in a grid search to tune the hyperparameter using logloss as my measurement of choice. I am implementing this in Python so Scikit-Learn as my tool of choice. The algorithm I settled on was GradientBoostingClassifier.

Problem

What I am interested in doing now is as I look at events in 2020, I am curious on how likely they are to succeed. When I look at my 2020 data I have the same 14 features (transform them the same way) and as well I instantly know if they succeeded or failed. So predicting Success/Failure is not interesting nor what I want to do. I can easily generate probabilities on this 2020 data, using my trained model, with predict_proba in sklearn.

Question

Now my partner wants me to apply this model on the same training data from 2016-2019. Initially this feels like a big no-go. You never predict your own training data as it will be heavily biased. But I am not predicting. These feels more like a traditional statistics descriptive problem where I look at the nature of known data and see how it behaves.

So again, I am not interesting in knowing/predicting if something WILL succeed or fail (we know this instantaneously). I am more interested in knowing if it succeeded (or failed) vs how likely it was to do so. A success with a 5% chance vs succeeding with 40% chance is way more interesting in this problem.

So my question, per the title is, can you apply a trained algorithm / model on its own training data if my interest is not in predicting forward but evaluating success vs likelihood of success (and still yield useful information)?

$\endgroup$
1
$\begingroup$

can you apply a trained algorithm / model on its own training data if my interest is not in predicting forward but evaluating success vs likelihood of success (and still yield useful information)?

Yes, absolutely. There are quite a few cases where applying a model on the training data is useful. The most common is probably to detect overfitting: a high difference in performance between the training and test set is a sign of overfitting. In general it can also be useful to know how the model performs on the training set in order to obtain an upper baseline for the performance.

Everybody says "don't predict on the training set" simply because it's a simple rule to remember and such an easy mistake to make for beginners. But as long as one understands what they are doing and knows that obviously the predictions obtained on the training set are biased, there's no problem.

The task described in the question makes sense to me, you have my permission to predict on the training set ;)

Just one more remark: there might be a bias in the training data itself, in the sense that if success for an event is very unlikely then the event will fail more often than suceed even when the condition of its success are satisfied. If the data contains this kind of cases, it means that the model is trained to predict "fail" for potentially successful features. Whether or not this impacts the model depends on whether there are enough similar but successful events in the data, I think.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.