Understanding correlation - Machine Learning

Question

I am experimenting a project on identifying cancer or not - Binary classification

The dataset has many columns. Here, I added correlation values between few input columns and the target column[cancer/healthy subjects] calculated using pandas pd.corr function

[('age', 0.043351857732763135),
 ('occupat', 0.012718551481541234),
 ('c1', 0.008842838683373164),
 ('c2', 0.008004032794076186),
 ('c3', 0.007534428151349621),
 ('cigar_stat', 0.007269967555618035),
 ...
 ...
]

The age is the only column having high correlation with output than all other input features and which is very less 0.04335. And still there are more features having correlation value in negative direction.

While training with ResNet18/50/Inception-Resnet V2, I go through many tunings, I got very less accuracy.

I understand from this experiments, if the correlation value is very less compared to the target_column, then the training accuracy is also not improving. - Is it actually a VALID point?

Having this dataset, I have no option to remove some columns which are all having very less correlation value with output, instead only use as it is.

What are the ways to handle this situation, so that I can get the good performance in training procedure?

Thanks

Answer 1

Unfortunately, things are not as simple.

About Correlation

For some simple models (especially linear / logistic regression), the correlation between feature and target variable is a good indicator whether a variable can improve the accuracy of the model (but also have a look at correlations between the variables).

In case of more complex models (such as neural networks), even features with zero correlation with the target variable can have a strong positive impact in the accuracy.

Look for example at these data points

If you only know the x1-coordinate of a point, there is nearly no information about the class variable. So the x1-value of a point is not correlated with the target variable. Similar, the x2-value has nor correlation with the target variable. But combining both allows for a very high accuracy when it comes predicting the color of a dot.

Putting this together: For your ResNet networks, the correlation of a feature variable with the target hat no strong impact on the accuracy

What else could be the reason?

To my understanding, you have tabular data with heterogeneous features (e.g. age and cigar_stat will have totally different values).

• In case of such features, preprocessing of the values is important for neural networks. Unfortunately, the right preprocessing can be a topic on its own and I will not be able to cover it, here.
• ResNet is made for images. In images, pixels have a relation based on there position (e.g. a strong difference betwen neighboring pixels might indicate an edge, ...). In tabular data, the relation given by the meaning of the variables, but not by there location in the table. In such a case, CNNs (which are included in ResNet ans imilar networks) do not make much sense.

In Summery: Without knowing your case in detail, I would assume that you are using the wrong model for your data. I would start with some tree-based methods (Random Forests, XGBoost, ...). They are typically well-suited for tabular data.