# Dropping features after final evaluation on test data

Would you please let me know if I am committing a statistical or machine learning mal-practice in this procedure?

I want to estimate meteorological variable y1 from $${x_1, ..., x_{10}}$$ variables. I use data from different weather stations. I keep some weather stations as test sites/data.

I do feature selection and hyper parameter tuning with cross-validation on training data. My model is Random Forest (RF) and two other tree based models.

Before I evaluate my models on test sites I was skeptic about keeping one of the features: Elevation of the weather station, $$x_{10}$$. This is a static feature that would be present/same in all rows of data related to a station. Knowing a tiny bit about RF made me worried that the model will use this as a kind of "site_id" and possibly overfit to this feature. It wouldn't make me worried if I was using linear/nonlinear regression models.

So I train my models once with and once without $$x_{10}$$ as a feature.

Then I evaluate my models on the test sites and turns out that the models without $$x_{10}$$ do significantly better on test sites.

Even before testing this hypothesis about the static feature, I wanted to do similar tests with dropping other features as well, say $$x_{9}$$.

Now my question is: now that I know that $$x_{10}$$ hurts my model, I like to retrain my models without $$x_{10}$$ and test model performance with and without $$x_{9}$$ in the filtered feature set.

To me it seems like Im using my test data to kind of filter out my feature so it is not right. But then, I have this information and if $$x_{10}$$ is hurting my models in the end, why should I go on testing hypothesis and preparing my models with $$x_{10}$$ being in them?