This question relates generally to all automated feature selection approaches. In my particular scenario, we have a python package called tsfresh and multiclass classification.

What has been done so far? I extracted features using tsfresh.extract_features without filtering any features. All those features are fed to RF model and the model is left to decide on important features for itself. Model performance is measured averaging cross-validation with 5 k-fold splits rather than using one single train-test split.

What I want to achieve? Following the official tsfresh documentation for multiclass selection, a reasonable thing to do would be to split the data before doing any feature selection using tsfresh.select_features. Since feature selection tends to be rather a demanding task and I have a lot of models, using CV with 5 k-fold splits increases the computation time tremendously. If I perform feature selection using tsfresh without any split, I am probably leaking data?

Do you have any alternative solutions to handle this scenario or do I have to get along with these heavy computation times coming with cross-validation? Do you think this is even necessary if I have RF model (because of the internal feature selection)?

  • $\begingroup$ I actually wonder who downvotes without saying anything. Must be some stingy complexes... $\endgroup$
    – Jumpman
    Commented Jun 20, 2022 at 18:52

2 Answers 2


You generally need to perform time aware CV if you want to avoid time leakage.

The questions wether you really need that time-aware CV, wether the computation time is woth it, if you need to perform feature selection before RF are entirely different questions, that don't have trivial answers. My general answer would be yes, yes, no, but there would be a lot of specific subcases where the answer would be different. If your computation time is the bottleneck, try to perform your calculations overnight.


Take a look at this generic example, and see if it does what you want.

from sklearn.datasets import load_boston
from sklearn.ensemble import RandomForestRegressor
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#Load boston housing dataset as an example
boston = load_boston()

X = boston["data"]
Y = boston["target"]
names = boston["feature_names"]
reg = RandomForestRegressor()
reg.fit(X, Y)
print("Features sorted by their score:")
print(sorted(zip(map(lambda x: round(x, 4), reg.feature_importances_), names), 

boston_pd = pd.DataFrame(boston.data)

boston_pd.columns = boston.feature_names

# correlations

features = boston.feature_names
importances = reg.feature_importances_
indices = np.argsort(importances)

plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='#8f63f4', align='center')
plt.yticks(range(len(indices)), features[indices])
plt.xlabel('Relative Importance')

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.