I'm using RapidMiner to evaluate correlation between attribute in my dataset. The problem is that a lot of values appear with '?'.

Someone can help me?

enter image description here

This is a sample of data enter image description here

  • $\begingroup$ I don't know the software, but it's likely that it fails to compute the correlation. That can be due to incomplete rows in the dataset, non-numeric data, zero division (in case of null variance, but that's not likely, given the number of cases), or something else. Can you share a sample of your data? $\endgroup$ – Romain Reboulleau Nov 20 '19 at 18:53
  • $\begingroup$ Yeah, I've edited the post. $\endgroup$ – Lorenzoi Nov 20 '19 at 21:27

All your data is non-numeric, so there is no straightforward method to compute a correlation value. It appears that the software does compute a correlation value between 2-valued fields (breast and irradiat).

You will have to work on your data if you want to compute the full correlation matrix:

  • Make fields with numeric values numeric. For instance, the 6th field seems to be numeric data encoded as strings (make sure those are not numbered categories that cannot be ordered, otherwise the correlation will mean nothing)
  • Give numeric values to ordered categories. For instance, for the age field, you could certainly use a centered numeric value for each category (45 instead of 40-49, 55 instead of 50-59, etc.), or just a rank.
  • Ignore rows with no data. For instance, in the 5th field, you have binary results (yes/no) and appear to have a few question marks. Removing the lines with question marks will make the field 2-valued, something that appears to allow the software to compute the correlation.

For the rest of the fields, there is no magic trick, unfortunately. You may want to one-hot-encode them, but interpreting the results will be more complicated. There are also ways to test categorical fields independence, but that's a different measure: see this post.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.