3
$\begingroup$

So, I'm working on a ML model that would have as potential predictors : age , a code for his city , his social status ( married / single and so on ) , number of his children and the output signed which is binary ( 0 or 1 ). Thats the initial dataset I have.

My prediction would be based on those features to predict the value of signed for that person.

I already generated a prediction on unseen data. Upon validation of the results with the predicted result vs the real data , I have 25% accuracy. While cross-validation gives me 65% accuracy. So I thought : Over-fitting

Here is my question, I went back to the early stages of the whole process, and started creating new features. Example : Instead of the code for the city which makes no sense to have that as input to a ML model, I created classes based on the percentage of signed, the city ,with the higher percentage of 'signed' ( the ouput ) , gets assigned to a higher value of class_city, which improved a lot in my Correlation Matrix the relationship signed-class_city which makes sense. Is what I'm doing correct? or shouldn't I create features based on the ouput? Here is my CM :

After re-modelling with 3 features only ( department_class , age and situation ) i tested my model on unseen data made of 148 rows compared to 60k rows in the training file.

First model with the old feature ( the ID of the departement ) gave 25% accuracy while the second model with the new feature class_department gave 71% ( Again on unseen data )

Note : First model with 25% has some other features as ID's ( they might be causing the model to have such a weak accuracy with the deparment_ID )

Improve this question
$\endgroup$
2
  • $\begingroup$ This is called "target encoding." As you and the answers note, the encoding should not involve your test set(s). Also possibly problematic are rare levels in the variable. $\endgroup$
    – Ben Reiniger ♦
    Mar 19, 2019 at 18:24
  • $\begingroup$ Thank you for your answer. my test set is untouched and un-used during the whole process if that's what you mean by not involving my test set $\endgroup$
    – Blenz
    Mar 20, 2019 at 8:21

2 Answers 2

Reset to default
3
$\begingroup$

You can create features based on output values, but you should be careful in doing this.

When you use the value of class_city (based on percentage of signed for that city) for a given data point, note that this calculation cannot include the current data point, since you will not have the value of ‘signed’ during prediction.

One way to handle this is to split the total data you have into three parts - estimation, train, test. The estimation set is used only to estimate the class_city values for each city. These values can then be used in the train and test data. This way, you have the label values without your model doing anything ‘unfair’. For testing, you can infact use the data from estimation+train sets to estimate the class_city values for use in the test set. The same holds true for any unseen data. You can use the class_city values estimated from all the previous data points.

In the context of time series data, for example, the class_city value for any data point can potentially use information from all previous data points, and should not use any information from future data points!

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your answer. I'm not quite familiar with the concept of data points, but this makes sense! $\endgroup$
    – Blenz
    Mar 20, 2019 at 8:29
  • 1
    $\begingroup$ This last part about future data is important. It's also important not to use data from the current date, as this would also be a data leak. $\endgroup$
    – Dan Carter
    Mar 20, 2019 at 12:28
  • $\begingroup$ Yes this is exactly what i was looking for ! $\endgroup$
    – Blenz
    Mar 22, 2019 at 17:01
3
$\begingroup$

No you should not do this, it is causing a data leak. Data leaks happen when the data you are using to train a machine learning algorithm happens to have the information you are trying to predict.

It will give your model information about your test data during training. It will cause your test scores to be overly optimistic and will make the model worse at generalizing to totally unseen data.

Good resource on data leakage

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for the answer, what would you recommend in case of ID's being given as input in a dataset? ( in the case of department_code ), what's a common approach for that kind of data to improve my model? $\endgroup$
    – Blenz
    Mar 19, 2019 at 15:34
  • $\begingroup$ Glad it helped! That is really a new question not directly related to this and is not really for the comments. If you feel that the question you asked here has been answered, mark this as correct. If you have new things you want answered you should open a new question. $\endgroup$
    – Simon Larsson
    Mar 19, 2019 at 16:17
  • $\begingroup$ Your comment makes sense, but my results contradict your answer, i'm waiting for maybe a better answer, if not, i'll mark your question as correct ( again it makes sense that i shouldn't give away information on the values i want to predict in features ) Note : I'm posting my results with the old feature and the new feature on unseen data! $\endgroup$
    – Blenz
    Mar 19, 2019 at 16:27
  • $\begingroup$ How does the result contradict my answer? You got higher score on both train and test set, which is what you expect from data leakage. Or is it something that I am missing? I am only addressing "shouldn't I create features based on the ouput?". $\endgroup$
    – Simon Larsson
    Mar 19, 2019 at 16:38
  • 2
    $\begingroup$ Yes, that is what I am saying. Data leakage gives you an overly optimistic test score. $\endgroup$
    – Simon Larsson
    Mar 19, 2019 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.