I am facing a weird problem in my on going project and thought if someone here could help me out with this. Actually I have large data set. I have to perform a regression task on top of that. While doing the initial analysis and feature selection task, I did correlation of all the features with the target variable. When I got the results I saw one of the feature having negative correlation with the target variable but my business unit suggests it must have a positive correlation to proceed further. Now they are asking me to filter out the rows causing negative correlation. I don't know how to proceed in this context. Can anybody help me out or give me heads up in this? Thank you very much in advance
First what comes to my mind: "the only statistics you can trust are the ones you have falsified yourself" - Churchill
However, to answer your question.
The correlation coefficient is defined as:
You can find data points that are negatively correlated by looking at the covariance, that is defined as:
First, you calculate the mean of your dependent and independent variable. Then you subtract the mean from the corresponding data points. After that, you multiply those two series. Negative values are rows that contribute to a negative correlation coefficient.
However, as a statistician, I am not supporting this approach (data deletion on purpose is not scientific working).