I am facing a weird problem in my on going project and thought if someone here could help me out with this. Actually I have large data set. I have to perform a regression task on top of that. While doing the initial analysis and feature selection task, I did correlation of all the features with the target variable. When I got the results I saw one of the feature having negative correlation with the target variable but my business unit suggests it must have a positive correlation to proceed further. Now they are asking me to filter out the rows causing negative correlation. I don't know how to proceed in this context. Can anybody help me out or give me heads up in this? Thank you very much in advance
$\begingroup$
$\endgroup$
asked Dec 9, 2019 at 15:30
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
First what comes to my mind: "the only statistics you can trust are the ones you have falsified yourself" - Churchill
However, to answer your question.
The correlation coefficient is defined as:
You can find data points that are negatively correlated by looking at the covariance, that is defined as:
First, you calculate the mean of your dependent and independent variable. Then you subtract the mean from the corresponding data points. After that, you multiply those two series. Negative values are rows that contribute to a negative correlation coefficient.
However, as a statistician, I am not supporting this approach (data deletion on purpose is not scientific working).
-
$\begingroup$ Thanks, actually my target variable is a probability score so would this approach work with that? I completely agree with your last line that data deletion should not done purposely but in my data every row corresponds a unique part of machine, so my BU suggested me to filter out the parts that are not significant. $\endgroup$ Commented Dec 9, 2019 at 16:15
-
$\begingroup$ Yes, as it is straight math, it works independent on the variable as long as it is continuous. $\endgroup$– Maeaex1Commented Dec 9, 2019 at 16:18