1
$\begingroup$

I am currently working on a binary classification problem using imbalanced data. The algorithm that I am using is random forest. The problem is about predicting whether each sales project will meet its target or not.

For example, a sales manager could have multiple sales project running under him. We need ML to predict what is the likelihood that each project will meet its target agreed during start of the project. Each projects runs for 3 to 5 year cycle. So, every year there is a specific target to be met.

Based on the year currently the project is in, we would like to know whether project will meet its target upto that specific year. If the project is in 3rd year, we need to find the likelihood for the project to meet its 1st 3 years target (1st, 2nd and 3rd year).

So, now my question is on including two columns/feature which contains the value of how much target achieved/units purchased till this time point (3rd year) as well as "target set at the start of the project". Is it okay to include the feature of "total target achieved/units purchased as on date" and "target set at the start of the project"?

or it is data leakage or considered biasing the model?

we have that target achieved/units purchased as on date info for every project which is updated frequently based on the purchase made.

Every project that we are trying to predict the likelihood, will either have achieved 0 % of the target or 10% of the target or 20% of the target or exceeded the target up to that time point etc. So, we have this info for all records.

And the output_label column is marked as 1 if they exceed the target and marked as 0 if they have not met the target. So, we feed the model the target set (ex:1000 units should be bought) for a project and also how much they have achieved as of now (ex: 200 units bought already) along with other variables.

So, do you think this is a data leakage or considered biasing the model? can I use these two features or not?

As I have the data for these two features at the start of my analysis itself. Meaning, if I am extracting data/building model today, I can find out what is the latest value for "target achieved as on date" yesterday and "target set at the start of the project" (using which labels are derived)

But what if ML model easily captures the relationship (if target achieved >= target set - high likelihood to meet the target else low likelihood to meet the target).

So, in this case do we need ML at all in the first place? Am confused. Of course, along with these features, am trying to few more input variables as well based on historical data. Can you guide me on whether incorporating these two features - target set and target achieved as of date is okay? But yes, including these features results in better performance of the model.

while these two features majorly drive the prediction to 87% of f1 in test data, if I include my additional features, they take upto 93% for f1 in test data. If I exclude these two features, f1 is about 55-60% for minority class.

But one thing, I found out was that these two columns are not heavily correlated within themselves and also with the target. So, am not sure how is prediction performance being increased so heavily after these two features

Also, important point to note is that my output variable is computed using a formula/rule that involves these two features.

However, when I validated the performance on the test data, I don't see any signs of overfitting or drop in performance. But yes, these two features drive the prediction all alone contributing to around 87% of f1 score where as other 3-4 predictors add another 5 points.

So, am I good to use these features in model building despite they being used to create rule-based label? I don't let the model know the exact formula/rule. So, what do you think?

$\endgroup$

2 Answers 2

2
$\begingroup$

Data leakage occurs in cases when you train a model with data that is not available for future testing/inference; or when you use same piece of data for training, and then for validation and/or testing. This short Kaggle article sums it up nicely.

If you have a feature (e.g. target_year_x) that somehow quantifies how much of the target goals are currently at year x achieved, I fear that this could introduce bias in your model, and may technically be data leakage. High values for that feature indicate that the project is close to meeting its goals, and is more likely to meet its target; thus the model would learn (the very obvious thing) that high values for target_year_x are highly predictive for the projects' success.

My suggestion is to maybe try multiple models, i.e., one model to predict success in first year, one in second, etc. Or, separate model for separate project phases, if you can somehow logically split the projects. If you try that, be careful not to include features that relate to latter phases for the earlier models (e.g., don't include features that provide information about the projects' second year performance, for the model that predicts in the first year).

Or, as the other answer by Brian Spiering suggests, which is also a good option IMO, you might want to consider to frame it as a time series prediction problem if you need multiple chronological predictions per project, rather than a binary classification one.

$\endgroup$
11
  • $\begingroup$ Isn't this problem similar to what we seen in sports? Ex: In football, when two teams are competing against each other, usually each team is given a likelihood to win the match..like team A has 70% and team B has 30%...To compute this likelihood, they make use of current goals scored, past win:loss ratio, home or away match etc...So, in this case if they could use the no of goals currently scored (and we can infer how much more goals are required to win the match), if we use it for our problem, is it a mistake? $\endgroup$
    – The Great
    May 15, 2023 at 23:08
  • $\begingroup$ If I don't use ML, how do I rank the likelihood of the projects to meet the target..Sales users may not have time to focus on all the projects. So, we need some mechanism to bring their attention to high likelihood to succeed projects and discard dead projects and not spend their resources, phone calls, follow up, customer visits etc...Instead of we manually coming up with rules and assigning weightages, using a simple model like Random Forest is not encouraged? I am asking only to understand when and when not to use ML. $\endgroup$
    – The Great
    May 15, 2023 at 23:10
  • $\begingroup$ Along with these two features, 'target set at start of project' and 'target achieved as of date', I see that inclusion of few more features results in improved performance of model by 3 points for auc and f1 ...can't this be the reason to justify use of ML? $\endgroup$
    – The Great
    May 15, 2023 at 23:12
  • $\begingroup$ Your sport analogy is spot on. I might be wrong about my claim there, had my doubts while typing the answer, but I am not fully convinced of it. Yes, the team scoring more goals is more likely to win, but wouldn't you say that's very obvious and somehow part of the target variable? Going back to the actual projects problem, if you have a way to quantify the project's success at a certain point in time, why not just sort the projects by their current success? Sales users can focus on the projects that are currently most successful. ML doesn't seem so necessary when phrased like that. $\endgroup$ May 15, 2023 at 23:26
  • $\begingroup$ ML should automate some part of your work. If without ML, you look at project properties X, Y, Z, etc., and make a decision whether someone should focus on it or not, and you want to (partly) automate this, then by all means use ML. Makes sure you have the correct data and mimic/replace your working process by model development. But if you can find a simpler solution, like what I mentioned about sorting in the other comment, and this makes sense, i.e., it works, then ML is not necessary. It's not a mistake, but you are ignoring a simpler solution $\endgroup$ May 15, 2023 at 23:29
2
$\begingroup$

One definition of data leakage is providing the model with data during training that would not be available at a future prediction time. The variable "total target achieved/units purchased as on date" is not data leakage according to that definition.

Your problem might be better framed as a time series prediction than a tabular prediction.

$\endgroup$
1
  • $\begingroup$ Thanks for the response May I check how can I frame this problem as time series prediction? Because I have one row for each project in my dataset amd each row contains info about the project from start date to current time of analysis.. $\endgroup$
    – The Great
    May 15, 2023 at 23:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.