Automated feature selection - Best practice to avoid data leakage?

Question

This question relates generally to all automated feature selection approaches. In my particular scenario, we have a python package called tsfresh and multiclass classification.

What has been done so far? I extracted features using tsfresh.extract_features without filtering any features. All those features are fed to RF model and the model is left to decide on important features for itself. Model performance is measured averaging cross-validation with 5 k-fold splits rather than using one single train-test split.

What I want to achieve? Following the official tsfresh documentation for multiclass selection, a reasonable thing to do would be to split the data before doing any feature selection using tsfresh.select_features. Since feature selection tends to be rather a demanding task and I have a lot of models, using CV with 5 k-fold splits increases the computation time tremendously. If I perform feature selection using tsfresh without any split, I am probably leaking data?

Do you have any alternative solutions to handle this scenario or do I have to get along with these heavy computation times coming with cross-validation? Do you think this is even necessary if I have RF model (because of the internal feature selection)?

Answer 1

You generally need to perform time aware CV if you want to avoid time leakage.

The questions wether you really need that time-aware CV, wether the computation time is woth it, if you need to perform feature selection before RF are entirely different questions, that don't have trivial answers. My general answer would be yes, yes, no, but there would be a lot of specific subcases where the answer would be different. If your computation time is the bottleneck, try to perform your calculations overnight.

Answer 2

Take a look at this generic example, and see if it does what you want.

from sklearn.datasets import load_boston
from sklearn.ensemble import RandomForestRegressor
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#Load boston housing dataset as an example
boston = load_boston()


X = boston["data"]
Y = boston["target"]
names = boston["feature_names"]
reg = RandomForestRegressor()
reg.fit(X, Y)
print("Features sorted by their score:")
print(sorted(zip(map(lambda x: round(x, 4), reg.feature_importances_), names), 
             reverse=True))


boston_pd = pd.DataFrame(boston.data)
print(boston_pd.head())

boston_pd.columns = boston.feature_names
print(boston_pd.head())

# correlations


features = boston.feature_names
importances = reg.feature_importances_
indices = np.argsort(importances)

plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='#8f63f4', align='center')
plt.yticks(range(len(indices)), features[indices])
plt.xlabel('Relative Importance')
plt.show()