what qualifies as a data leakage?

Question

I am currently working on a binary classification problem using imbalanced data. The algorithm that I am using is random forest. The problem is about predicting whether each sales project will meet its target or not.

For example, a sales manager could have multiple sales project running under him. We need ML to predict what is the likelihood that each project will meet its target agreed during start of the project. Each projects runs for 3 to 5 year cycle. So, every year there is a specific target to be met.

Based on the year currently the project is in, we would like to know whether project will meet its target upto that specific year. If the project is in 3rd year, we need to find the likelihood for the project to meet its 1st 3 years target (1st, 2nd and 3rd year).

So, now my question is on including two columns/feature which contains the value of how much target achieved/units purchased till this time point (3rd year) as well as "target set at the start of the project". Is it okay to include the feature of "total target achieved/units purchased as on date" and "target set at the start of the project"?

or it is data leakage or considered biasing the model?

we have that target achieved/units purchased as on date info for every project which is updated frequently based on the purchase made.

Every project that we are trying to predict the likelihood, will either have achieved 0 % of the target or 10% of the target or 20% of the target or exceeded the target up to that time point etc. So, we have this info for all records.

And the output_label column is marked as 1 if they exceed the target and marked as 0 if they have not met the target. So, we feed the model the target set (ex:1000 units should be bought) for a project and also how much they have achieved as of now (ex: 200 units bought already) along with other variables.

So, do you think this is a data leakage or considered biasing the model? can I use these two features or not?

As I have the data for these two features at the start of my analysis itself. Meaning, if I am extracting data/building model today, I can find out what is the latest value for "target achieved as on date" yesterday and "target set at the start of the project" (using which labels are derived)

But what if ML model easily captures the relationship (if target achieved >= target set - high likelihood to meet the target else low likelihood to meet the target).

So, in this case do we need ML at all in the first place? Am confused. Of course, along with these features, am trying to few more input variables as well based on historical data. Can you guide me on whether incorporating these two features - target set and target achieved as of date is okay? But yes, including these features results in better performance of the model.

while these two features majorly drive the prediction to 87% of f1 in test data, if I include my additional features, they take upto 93% for f1 in test data. If I exclude these two features, f1 is about 55-60% for minority class.

But one thing, I found out was that these two columns are not heavily correlated within themselves and also with the target. So, am not sure how is prediction performance being increased so heavily after these two features

Also, important point to note is that my output variable is computed using a formula/rule that involves these two features.

However, when I validated the performance on the test data, I don't see any signs of overfitting or drop in performance. But yes, these two features drive the prediction all alone contributing to around 87% of f1 score where as other 3-4 predictors add another 5 points.

So, am I good to use these features in model building despite they being used to create rule-based label? I don't let the model know the exact formula/rule. So, what do you think?

Answer 1

Data leakage occurs in cases when you train a model with data that is not available for future testing/inference; or when you use same piece of data for training, and then for validation and/or testing. This short Kaggle article sums it up nicely.

If you have a feature (e.g. target_year_x) that somehow quantifies how much of the target goals are currently at year x achieved, I fear that this could introduce bias in your model, and may technically be data leakage. High values for that feature indicate that the project is close to meeting its goals, and is more likely to meet its target; thus the model would learn (the very obvious thing) that high values for target_year_x are highly predictive for the projects' success.

My suggestion is to maybe try multiple models, i.e., one model to predict success in first year, one in second, etc. Or, separate model for separate project phases, if you can somehow logically split the projects. If you try that, be careful not to include features that relate to latter phases for the earlier models (e.g., don't include features that provide information about the projects' second year performance, for the model that predicts in the first year).

Or, as the other answer by Brian Spiering suggests, which is also a good option IMO, you might want to consider to frame it as a time series prediction problem if you need multiple chronological predictions per project, rather than a binary classification one.

Answer 2

One definition of data leakage is providing the model with data during training that would not be available at a future prediction time. The variable "total target achieved/units purchased as on date" is not data leakage according to that definition.

Your problem might be better framed as a time series prediction than a tabular prediction.