1
$\begingroup$

I'm new in data analysis area and didn't have very strong sattistical background...

Now, I'm trying to filter out those numeric columns which have high correlation. At this moment, I don't plan to use dimensional reduction algorithms such as PCA because I'm still collecting data, and there are too many columns, I hope to remove some columns first before linking different sources of data together.

Playing with PCA and machine learning models will be after linking those sources of data.

So, at this "before" stage, I'm trying to use R library caret, findCorrelation() method, but I could not find how does this method work.

Now my question is, I should use this method after data cleaning (such as, deal with missing data, outliers, data imbalance) or it doesn't matter to do it before or after?

$\endgroup$
  • $\begingroup$ Depends a whole lot on what the data is like, before and after. $\endgroup$ – Sean Owen Sep 29 '16 at 10:43
3
$\begingroup$

In most cases, you will want to run findCorrelation after cleaning. The reason being the cleaning process is either removing unwanted data points (that you deem irrelevant anyway) or computing some transformation that you'll want findCorrelation to be aware of.

Now, that said, I've often run it twice, once before cleaning and once after if my transformation could alter the correlation structure of the data.

Fortunately, since you're using caret, it is a simple matter to estimate your models both ways, then compare the ultimate differences in accuracy. Unfortunately, the correct answer to many questions in ML is 'do it both ways then compare'

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.