I used to tried making a fake correlation in my mocking dataset and found that if the score is more than 0.5 I can reduce feature to avoid singularity

In the given example they are many correlated variables. Therefore I am thinking about dimensionality reduction. The one feature here that I plan to remove is Hum-6D05 because it has strong correlation between Hum-6D12, Hum-6F04, and Hum-6F14

Am I correct?
Any comment or advice are welcomed

enter image description here


2 Answers 2


That isn't easy to answer, as it highly depends on your taks. The only time where it is definitely recommended to remove a feature is when two attributes correlate perfectly (e.g. price and tax of a product).

Removing attributes that do not correlate perfectly could worsen your accuracy as information gets lost. But keeping them could affect the computation time. If you don't worry about computation time, I would highly recommend you to keep every bit of information about the data.

If you are concerned about overfitting there are way better techniques such as regularization to solve this problem.

Note: Dimensionality reduction is mostly used to visualize high dimensional data, so that a human eye can detect patterns. For example by shrinking the dimension to 3D or even 2D one could observe an exponential dependency.

  • $\begingroup$ Interesting! Thank you for your answer. IMO temperature and humidity might has a direct correlation. That's why I would like to confirm my plan before proceed. $\endgroup$
    – ii2
    Dec 15, 2018 at 16:28
  • $\begingroup$ Unfortunately I can't comment on the answer above, so I would like to address, that that answer isn't quite correct. Correlation != Collinearity. Multicollinearity only belongs to multiple regression. If your model uses something different than multiple regression don't look for multicollinearity. Look at quora.com/… for correlation vs. collinearity $\endgroup$
    – oezguensi
    Dec 15, 2018 at 18:14

The formal name for your concern is "multicollinearity". This can be a concern in statistically based techniques. There are formal tests that you can perform to assess multicollinearity like Variation Inflation Factor.

That said, many machine learning techniques are indeed not that sensitive to multicollinearity, so it may not matter if for example, you were training a neural network.

To answer your direct question, the terms you highlight are probably not so correlated that I would remove them.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.