# I do feature engineering on the full dataset, is this wrong?

I am aiming to predict the number of days it takes to sell a given property, let's call this variable "DaysForSale" - in short DfS

Using the DfS I created a variable called "median_dfs_grouped_street_name" which returns the median days it takes to sell a property for the different streets available in the dataset. (The street names are all categorized).

After this, I do my train/test split and run my Random Forest method.

Using the feature_imporatances function I see that the new feature is the second most important, which makes me wonder if this is the correct approach?

I have two questions:

1. Is it wrong to develop features using the target variable?
2. Is it wrong to do feature engineering on the full dataset?

Is it wrong to develop features using the target variable?

Not necessarily. It is called "target encoding" or "Mean encoding" and can be very useful. In your case you could, for example, use the DfS of your train data to calculate a median value per street. But you need to carefully design the target encoding to avoid overfitting (there are different strategies to do that - see below link). And for the test data you can only use the target encoding based on your train data.

The Coursera course "How to Win a Data Science Competition: Learn from Top Kagglers" has great content on target/mean encoding to be found here.

Is it wrong to do feature engineering on the full dataset?

Not necessarily. As pointed out in Nicolas' answer you need to be careful to not leak data though.

Here's an example where it would be ok: let's assume one of your features is date of enlisting which is the date when the property was published for sale. You could, for example, add a feature to the whole dataset called days since enlisting which simply calculates the days between now and when the property was published for sale. However, your median is an example which results in data leakage since it is not "per row" data engineering but "across rows" data engineering applied to train and test data.

That's why the safer approach is to first split the data, remove the target variable from the val/test data and then do feature engineering. Thereby, you avoid any unintended data leakage.

• Exactly what I thought. So I would take the median of the DfS from the training dataset, and add it to the test data, and not calculate the median for the test dataset as well? – doomdaam Aug 25 '20 at 10:02
• @doomdaam Yes, but you need to test if that very straight forward approach leads to overfitting (and I assume it does). And if it does you need try out a different strategies for target encoding. One strategy to do that is k-fold target encoding: This means to split your train data into, for example, k=5 folds. Then you calculate the median for a given property only based on the target variables from the other 4 folds. Alternatively, you could apply a rolling target encoding. That is, to calculate the median for a property in your train data only based on the previous rows in your dataframe. – Sammy Aug 25 '20 at 10:14
• The random forest bootstraps on columns (variables) as well as rows (samples) so it can look at the two columns separately. Personally I prefer a tool like Boruta which uses a little bit of stats mojo to independently evaluate all the columns and reasonably indicate which ones are unlikely to be informative. – EngrStudent Aug 27 '20 at 17:57

You’re correct: you should avoid feature engineering that brings information about the whole data set, including the testing data, into the training data set.

By involving your test data in the calculation of a median that is then available in your training data, you are leaking information from the testing data set into the training data set.

• Thanks for the fast answer! Another question, could I create the feature using the target variable but after the test/train split? I would argue, the feature is saying something about how the properties on a street had sold previously - without interfering with the dependent variable. – doomdaam Aug 25 '20 at 8:45
• Sadly not! You can only create features using information that will be available to you when you deploy the model in the real world. The thing you’re trying to predict won’t be known, otherwise you wouldn’t be trying to predict it :) – Nicholas James Bailey Aug 25 '20 at 9:13
• For this very specific project I'm working on of predicting days for sale: Is it wrong to use the feature I created? I would assume this is where domain knowledge comes into play. When trying to predict how fast a property will be sold, you can usually look at nearby properties and see how fast they have been sold, this should give you an idea of the potential of the property. And this is still forbidden to use? :) – doomdaam Aug 25 '20 at 9:26
• If you calculate it in a way that will be possible “in the wild”, for example by calculating the average data until sale for houses on the street that had sold previously (i.e. not including houses that hasn’t yet sold in your calculation) – Nicholas James Bailey Aug 25 '20 at 13:18
• I went with this approach: kaggle.com/ogrellier/…. Target encoding based on the street and days for sale only based on the training data. It does a rolled average, and finaly adds it to the test dataset making it possible to use for prediction. – doomdaam Aug 25 '20 at 13:31