Before answering your question I would ask if you actually need so many variables or you can apply some dimensionality reduction technique (i.e. PCA)? It could boost your training speed and maybe result in a better model efficiency.
Lets suppose for some reason (i.e. model explainability) you need the exact variables. You can still remove some of them without using feature importance (i.e. removing highly correlated ones, but there are other alternatives too). Indeed using only feature importance to remove variables can be dangerous. Imagine there is an important factor that highly influence your model but it shows up in several variables. In this case it can happen that the importance of this factor will be distributed among those variables. You will experience maybe that those variables are less important and remove them all thereby also eliminating an otherwise important factor.
To answer your question. I would before step 1 take away my test set and only use the training set for any kind of modeling activity. It's a good practice to guarantee that your test set has never been seen by the model and it did not influence it in any way.
In this case you can just remove less important features and measure the final results on the test set, it's not necessary to do CV.
On the other hand you will probably face with the challenge how to define your threshold for eliminating variables based on feature importance.You can try to do some gridsearch on it. In this case you should avoid measuring the results of different models on the test set or you risk over-fitting on it. In this case using CV can be a good solution to define your threshold.
