What to do when we can't trust our human classifiers?

Question

Suppose we want to design a neural network that can diagnose skin cancer. We want this neural network to consider the possibility that the doctor we hired misclassified some of our images while labeling. How can we design our neural network?

Answer 1

First, neural networks are good in dealing with"label noise". I'm currently on mobile/vacation, so remind me to search the paper on Friday.

Second, the more important question is how to get a good ground truth. Without a good ground truth you can't evaluate your models, no matter how good they might be.

I see the ways:

(1) have multiple experts label the stuff. Then you can make the ground truth a probability, not a simple label. If 9 experts say it is cancer and 1 says it is not, you would label it with 90%

(2) wait. If you can access the patients data, it will likely be more obvious in a year (especially if it was not treated)

(3) other diagnostic methods: I'm not a medical doctor, but I'm pretty sure there are invasive methods to diagnose cancer which are reliable

Answer 2

I have the following solutions:

• If you have abundant data you can shuffle them and make validation and training data. After that, your neural network should exploit generalization techniques not to overfit the training data. By doing so, you may have relatively acceptable performance which works in noisy situations.
• The other technique is evaluating Bayes error. This does not have any relation to the neural nets. It just tries to investigate in the feature space of the problem what percentage of your data is misleading, having same input patterns with contradictory labels.
• Another approach can be using an existing model for validating the data-set.