I am currently working on a binary classification problem using imbalanced data. The algorithm that I am using is random forest. The problem is about predicting whether each sales project will meet its target or not.
For example, a sales manager could have multiple sales project running under him. We need ML to predict what is the likelihood that each project will meet its target agreed during start of the project. Each projects runs for 3 to 5 year cycle. So, every year there is a specific target to be met.
Based on the year currently the project is in, we would like to know whether project will meet its target upto that specific year. If the project is in 3rd year, we need to find the likelihood for the project to meet its 1st 3 years target (1st, 2nd and 3rd year).
So, now my question is on including two columns/feature which contains the value of how much target achieved/units purchased till this time point (3rd year) as well as "target set at the start of the project". Is it okay to include the feature of "total target achieved/units purchased as on date" and "target set at the start of the project"?
or it is data leakage or considered biasing the model?
we have that target achieved/units purchased as on date info for every project which is updated frequently based on the purchase made.
Every project that we are trying to predict the likelihood, will either have achieved 0 % of the target or 10% of the target or 20% of the target or exceeded the target up to that time point etc. So, we have this info for all records.
And the output_label column is marked as 1 if they exceed the target and marked as 0 if they have not met the target. So, we feed the model the target set (ex:1000 units should be bought) for a project and also how much they have achieved as of now (ex: 200 units bought already) along with other variables.
So, do you think this is a data leakage or considered biasing the model? can I use these two features or not?
As I have the data for these two features at the start of my analysis itself. Meaning, if I am extracting data/building model today, I can find out what is the latest value for "target achieved as on date" yesterday and "target set at the start of the project" (using which labels are derived)
But what if ML model easily captures the relationship (if target achieved >= target set - high likelihood to meet the target else low likelihood to meet the target).
So, in this case do we need ML at all in the first place? Am confused. Of course, along with these features, am trying to few more input variables as well based on historical data. Can you guide me on whether incorporating these two features -
target set and
target achieved as of date is okay? But yes, including these features results in better performance of the model.
while these two features majorly drive the prediction to 87% of f1 in test data, if I include my additional features, they take upto 93% for f1 in test data. If I exclude these two features, f1 is about 55-60% for minority class.
But one thing, I found out was that these two columns are not heavily correlated within themselves and also with the target. So, am not sure how is prediction performance being increased so heavily after these two features
Also, important point to note is that my output variable is computed using a formula/rule that involves these two features.
However, when I validated the performance on the test data, I don't see any signs of overfitting or drop in performance. But yes, these two features drive the prediction all alone contributing to around 87% of f1 score where as other 3-4 predictors add another 5 points.
So, am I good to use these features in model building despite they being used to create rule-based label? I don't let the model know the exact formula/rule. So, what do you think?