This question relates generally to all automated feature selection approaches. In my particular scenario, we have a python package called tsfresh and multiclass classification.
What has been done so far? I extracted features using tsfresh.extract_features without filtering any features. All those features are fed to RF model and the model is left to decide on important features for itself. Model performance is measured averaging cross-validation with 5 k-fold splits rather than using one single train-test split.
What I want to achieve? Following the official tsfresh documentation for multiclass selection, a reasonable thing to do would be to split the data before doing any feature selection using tsfresh.select_features. Since feature selection tends to be rather a demanding task and I have a lot of models, using CV with 5 k-fold splits increases the computation time tremendously. If I perform feature selection using tsfresh without any split, I am probably leaking data?
Do you have any alternative solutions to handle this scenario or do I have to get along with these heavy computation times coming with cross-validation? Do you think this is even necessary if I have RF model (because of the internal feature selection)?