Why is an ML algorithm performing better with correlated features, than the one with uncorrelated ones?

Question

I have a dataset with all numerical values. Since the features were not many, I created more by multiplying pairs of each other. This created some highly correlated features, as expected. Now, I created a pipeline as below:

features_preprocessor= ColumnTransformer(transformers=[('numeric', num_transformer, [ 'f1','f2', 'f3', 'f4', 'f5', 'f6', 'f7',...., 'f26'])], remainder='passthrough')
pipe= Pipeline(steps=[
    ('preprocessor', features_preprocessor),
    ('regg', RandomForestRegressor())
])
xtr,xte,ytr,yte= train_test_split(x,y,test_size= 0.3)
pipe.fit(xtr,ytr)
ypred= pipe.predict(xte)
print("MAE",mean_absolute_error(yte,ypred))

This gave an MAE of 365, which is really good, considering my target variable ranges from 60000 - 110000. Note that I am not transforming my target variable, so the scale remains constant when comparing MAE values.

But then I removed the correlated features as below:

corr_features=set()
for i in range(len(highcorr.columns)):
    for j in range(i):
        if abs(highcorr.iloc[i,j])>0.8:
            colname= highcorr.columns[i]
            corr_features.add(colname)
cleandata=data.drop(corr_features, axis=1)

Now when I train the same pipeline, I get an MAE of 1023. I also tried splitting the data into training and testing first and found the correlated features only using the training data. Then removed those features from both training and testing data. This gave an MAE of 1072, which is worse than before, but understandable. I was expecting the results to get better, as multicollinearity causes fluctuating coefficients. Is my understanding wrong?

Answer 1

Few points to not here:

1. Multicollinearity effects linear model much more as compared to Random forest as it is picking up different set of features (read sampling with replacement) for every model and every model/tree see different data points. Feature importance may be impacted a little by multicollinearity

2. Multicollinearity does not impact model performance negatively but effects interpretation of feature importance

3. When we add polynomial features we are increasing the model complexity model starts performing better but be cautious about overfitting

So, its perfectly fine and expected that model performance is increasing as you are introducing polynomial features but be very cautious about overfitting and while interpreting feature importance