What are the conditions to remove highly correlated features in a correlation map? Given the correlation map below, is it OK to remove diagnosis feature and should we remove highly correlated features in every case.
How would you define highly correlated?
Normally one would decide on the threshold, of say Pearson's correlation coefficient. When the magnitude of Pearson's correlation coefficient would be above this value, you would call the two features correlated.
The above would help you to look for pairwise correlation. To detect correlation between several features at once, you could look at Fraction of Variance Unexplained or, equivalently, at Variance Inflation Factor.
It may also be a good idea to look at the kernel of the correlation matrix, i.e. the space spanned by eigenvectors of the matrix that correspond to small eigenvalues. Technically kernel corresponds to zero eigenvalues, but practically it is better to define it to correspond to small eigenvalues. Which of your features have large projections into the kernel space? Those should be removed first. Essentially removing correlated features is equivalent to making correlation matrix non-singular, which is equivalent to reducing the dimension of its kernel to zero, i.e. having no eigenvectors with zero eigenvalues.