Timeline for Dataset with some mislabeled data (around 1%)
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 17, 2020 at 13:05 | vote | accept | alpuy | ||
| Nov 16, 2020 at 1:14 | comment | added | Erwan | Yep! That's the most costly but also safest option, since it lets you evaluate the correction method on a random sample. | |
| Nov 16, 2020 at 1:06 | comment | added | alpuy | By re-annotate you mean check manually every one of the 1000 instances and correct the ones that are wrong? So we will have 1000 instances with 100% correct labels? | |
| Nov 14, 2020 at 12:18 | comment | added | Erwan | ... around 1% of instances which have been changed. In this case the manual analysis is only to check how many of these instances predicted with a different label were actually errors in the original label: if for instance 90% of these instances were errors which are corrected by the new predicted label, that's good. But if more than 50% of these instances were originally correct, then the automatic re-annotation does more harm than good. | |
| Nov 14, 2020 at 12:14 | comment | added | Erwan | @alpuy I'm not sure I understand your solution but I don't think that's the same as what I proposed. In my first option you would need to manually re-annotate at least around 1000 instances, because you need to see what happens with at least 10-20 cases of errors and 1% of 1000 = 10. This option lets you analyze all the possible cases after training/testing. In my second option you just run the training/testing using existing labels, then after that you analyze only the cases where the predicted label is different from the original label. If the system works well, you should have only ... | |
| Nov 13, 2020 at 13:16 | comment | added | alpuy | I like your answer because it goes in the direction that i was thinking as a validation for this method. Let me see if i get what you are trying to say. For example lets assume that we have 1000 samples. I should pick lets say 50 and change the label. Then after training the algorithm i should check how the 50 samples get labeled, the algorithm will be fine if from the 1000 samples, about 50 get their label changed and with the value of the previous label before changing. Is this what you mean? Thanks! | |
| Nov 11, 2020 at 23:54 | history | answered | Erwan | CC BY-SA 4.0 |