0
$\begingroup$

I am facing a weird problem in my on going project and thought if someone here could help me out with this. Actually I have large data set. I have to perform a regression task on top of that. While doing the initial analysis and feature selection task, I did correlation of all the features with the target variable. When I got the results I saw one of the feature having negative correlation with the target variable but my business unit suggests it must have a positive correlation to proceed further. Now they are asking me to filter out the rows causing negative correlation. I don't know how to proceed in this context. Can anybody help me out or give me heads up in this? Thank you very much in advance

$\endgroup$
0
$\begingroup$

First what comes to my mind: "the only statistics you can trust are the ones you have falsified yourself" - Churchill

However, to answer your question.

The correlation coefficient is defined as:

enter image description here

You can find data points that are negatively correlated by looking at the covariance, that is defined as:

enter image description here

First, you calculate the mean of your dependent and independent variable. Then you subtract the mean from the corresponding data points. After that, you multiply those two series. Negative values are rows that contribute to a negative correlation coefficient.

However, as a statistician, I am not supporting this approach (data deletion on purpose is not scientific working).

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, actually my target variable is a probability score so would this approach work with that? I completely agree with your last line that data deletion should not done purposely but in my data every row corresponds a unique part of machine, so my BU suggested me to filter out the parts that are not significant. $\endgroup$ – shivanshu dhawan Dec 9 '19 at 16:15
  • $\begingroup$ Yes, as it is straight math, it works independent on the variable as long as it is continuous. $\endgroup$ – Maeaex1 Dec 9 '19 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.