Should you look at the correlation between data points before using SVM to classify the data? Why?

Question

Total beginner here, so apologies if this comes off as a wild question.

I am learning how to classify data in SVM. I saw an example on Kaggle where someone had included a correlation matrix on the data points prior to classifying it. They didn't explain why.

And so, I'm wondering...is there any merit to doing a correlation analysis on the data prior to classifying it?

Several follow-ups to that as well:

1. Does it (the correlation) show in any way that your data is appropriate for classification? And if not, is it possible to accommodate for that?
2. If you know that two points have a very high correlation, is that a sign that the data set might be very good at classifying some of the data that you have, but not all of it?
3. Will items that have low correlation act as "noise"?

Thank you!!

Answer 1

Correlation analysis is very useful for a number of reasons, and not just for SVM classification.

In the case of feature selection, if two features in your dataset are highly correlated it means there is redundancy in your feature set (i.e. both features are explaining similar sources of variation in your data). By eliminating redundant features you can reduce the complexity of your model, since you will have fewer parameters to optimise.

It can also inform feature extraction. If you find there is a lot of correlation amongst your features, then your dataset may have a lower effective dimensionality than the current feature set you're working with. A technique such as PCA might allow you to extract a new smaller set of features that capture most of the variance in your original feature set, again eliminating redundant features and reducing the number of parameters that must be optimised (IMPORTANT: Methods like PCA must be performed on normalised data, otherwise features with larger value ranges will be prioritised).

This can also act as a form of regularisation - removing noise-based variance from your data to avoid model overfitting.

As for your follow-up questions:

1. Sure, as mentioned above, it can show which features in your dataset are redundant and therefore probably don't need to be included for the classification task.

2. As mentioned above also, this would be a sign that there is redundancy between those two features, and you may want to only use one.

3. No, features that are poorly correlated with other features are not necessarily noise.

While I'm sure there are additional reasons beyond those listed here, this is hopefully enough to demonstrate the importance of performing correlation analysis when taking on a classification (and regression) task!