If there are two continuous independent variables that show a high amount of correlation between them, can we remove this correlation by multiplying or dividing the values of one of the variables with random factors (E.g., multiplying the first value with 2, the second value with 3, etc.). We would be keeping a copy of the original values of the independent variable whose values have been transformed for proper comparison of the original values with the prediction results received. Can this be considered as an alternative to dropping one of the highly correlated variables?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ I am not sure I quite understand what you are saying. If the variables are independent then the correlation between them will be zero (see Wikipedia's article on correlation and dependence). Adding noise to the data (as you propose to do) will most likely be detrimental to the performance of any model you create based on that data. Is there any reason in particular you do not want to drop one of the two correlated variables from your model? $\endgroup$– RenéCommented Apr 18, 2021 at 8:01
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
High correlation between 2 features, eg $x_1$ and $x_2$ means that there is a linear relation between the two features (ie one is a linear transformation of the other), $x_2 = c_0 + s \cdot x_1 + \epsilon$.
That means any linear transformation of one or both features (eg multiplying by random factors), simply leaves the linear relation intact.
$x_{21}$ = $a \cdot x_2$, $x_{11} = b \cdot x_1$
Then
$x_{21} = a \cdot c_0 + a \cdot s/b \cdot x_{11} + \epsilon'$
Thus again there is linear relation thus high correlation.
Only non-linear transforms can alter the linear relation, but these can affect negatively the outcome.
Once a very strong correlation exists between 2 features then one of them is simply discarded, and only one remains, as it does not offer any new information by itself.
See also: https://datascience.stackexchange.com/a/87653/100269
$\begingroup$
$\endgroup$
can we remove this correlation by multiplying or dividing the values of one of the variables with random factors (E.g., multiplying the first value with 2, the second value with 3, etc.)
No you can not decorrelate 2 variables by multiplying or dividing them by a random factors, you can use one of the following methods instead if you want to keep the 2 variables:
- Divide a and b:
new_a = a / b - Powering up a:
new_a = a^2, a^3, ... - PCA
- ...
$\begingroup$
$\endgroup$
You can use PCA to transform your N correlated features to N uncorrelated features.
You can then check for correlation between your N uncorrelated features against your target and choose to drop the ones with very low correlation.
