I have a dataset with all numerical values. Since the features were not many, I created more by multiplying pairs of each other. This created some highly correlated features, as expected. Now, I created a pipeline as below:
features_preprocessor= ColumnTransformer(transformers=[('numeric', num_transformer, [ 'f1','f2', 'f3', 'f4', 'f5', 'f6', 'f7',...., 'f26'])], remainder='passthrough') pipe= Pipeline(steps=[ ('preprocessor', features_preprocessor), ('regg', RandomForestRegressor()) ]) xtr,xte,ytr,yte= train_test_split(x,y,test_size= 0.3) pipe.fit(xtr,ytr) ypred= pipe.predict(xte) print("MAE",mean_absolute_error(yte,ypred))
This gave an MAE of 365, which is really good, considering my target variable ranges from 60000 - 110000. Note that I am not transforming my target variable, so the scale remains constant when comparing MAE values.
But then I removed the correlated features as below:
corr_features=set() for i in range(len(highcorr.columns)): for j in range(i): if abs(highcorr.iloc[i,j])>0.8: colname= highcorr.columns[i] corr_features.add(colname) cleandata=data.drop(corr_features, axis=1)
Now when I train the same pipeline, I get an MAE of 1023. I also tried splitting the data into training and testing first and found the correlated features only using the training data. Then removed those features from both training and testing data. This gave an MAE of 1072, which is worse than before, but understandable. I was expecting the results to get better, as multicollinearity causes fluctuating coefficients. Is my understanding wrong?