I have a dataset with at least 70% of labels incorrect.
I'd expect that incorrect labels would compensate each other while true labels will be taught properly (given a very large dataset).
For example, if I have 300 samples saying
a => -1 and 300 samples saying
a => 1, the result for the input "a" eventually will be 0 (for a regression problem).
If I use Adam for the example above, won't it affect the results for the inputs with noisy labels due to its adaptive nature? Won't it be better to use SGD instead and decay the learning rate, or does Adam change its weights only at the end of every epoch?