# Automated feature selection - Best practice to avoid data leakage?

This question relates generally to all automated feature selection approaches. In my particular scenario, we have a python package called tsfresh and multiclass classification.

What has been done so far? I extracted features using tsfresh.extract_features without filtering any features. All those features are fed to RF model and the model is left to decide on important features for itself. Model performance is measured averaging cross-validation with 5 k-fold splits rather than using one single train-test split.

What I want to achieve? Following the official tsfresh documentation for multiclass selection, a reasonable thing to do would be to split the data before doing any feature selection using tsfresh.select_features. Since feature selection tends to be rather a demanding task and I have a lot of models, using CV with 5 k-fold splits increases the computation time tremendously. If I perform feature selection using tsfresh without any split, I am probably leaking data?

Do you have any alternative solutions to handle this scenario or do I have to get along with these heavy computation times coming with cross-validation? Do you think this is even necessary if I have RF model (because of the internal feature selection)?

• I actually wonder who downvotes without saying anything. Must be some stingy complexes... Jun 20 at 18:52

Take a look at this generic example, and see if it does what you want.

from sklearn.datasets import load_boston
from sklearn.ensemble import RandomForestRegressor
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#Load boston housing dataset as an example

X = boston["data"]
Y = boston["target"]
names = boston["feature_names"]
reg = RandomForestRegressor()
reg.fit(X, Y)
print("Features sorted by their score:")
print(sorted(zip(map(lambda x: round(x, 4), reg.feature_importances_), names),
reverse=True))

boston_pd = pd.DataFrame(boston.data)

boston_pd.columns = boston.feature_names

# correlations

features = boston.feature_names
importances = reg.feature_importances_
indices = np.argsort(importances)

plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='#8f63f4', align='center')
plt.yticks(range(len(indices)), features[indices])
plt.xlabel('Relative Importance')
plt.show()