Is it okay to create a derived model of highly correlated features?

Question

In my data set 2 features C1 and C2 are highly correlated. I did the following steps. Could you please let me know if it is correct and make sense? do you have a better approach?

First I used the linear model to find the fitted line: C1=a*C2+b

from sklearn import linear_model
reg=linear_model.LinearRegression()
y_reg=data1['C1']
x_reg=data1['C2']
reg.fit(x_reg2,y_reg2)
a=reg.coef_
b=reg.intercept_
print(a,b)

After finding a and b I removed C1 and C2 from the dataset and added a new Variable: new=a*C1+b

My next question is how I can understand if this line is good?

Answer 1

No I don't think this is good: this means that you replace both variables with only an estimated value of C2 based on C1 (or of C1 based on C2, it's not totally clear to me).

Assuming you really want to avoid having two highly correlated features, you could instead:

• just arbitrarily pick one of them (which is similar to what you're doing but "cleaner")
• simply use the average of the two values (which might be similar to what you intended to do?)

Over a whole set of features you could use more advanced methods such as feature extraction.

But it's important to make sure you actually need to do this: correlated features are a problem for some ML methods, but not all of them. For example having correlated features with Decision Trees wouldn't cause any problem. In any case it's worth experimenting with/without the feature in order to observe what actually happens with your data.