What is missing from the following Curriculum Learning implementation in a Deep Neural Net?

Question

First of all we have a classification task.
So we use the typical softmax cross entropy to classify.

Current implementation of curriculum learning is as follows.

• First we train our best version of the neural net
• At the last epoch we get all of the cross entropies for all the instances. So smaller cross entropies are better classified and larger ones not so well.
• Then we sort the instances based on the cross entropies.
• Then we start training the instances from easier to harder as the curriculum learning theory suggests

Note that we have already experimented with various steps and repetitions. So in one example we took the first 200 batches and trained them two times before going to the next batch and so on until an epoch is completed.
In another example we took the first 10 batches and trained them only once and then the next 10 and the next 10 and so on until the end of the epoch.

All of the experiments so far have concluded that the neural network is has a relatively ok accuracy at the beginning and this gets worsen as the more difficult instances come along. The final accuracy is much worse than expected and in addition the maximum accuracy is still quite bad.

Why is this curriculum learning not working? Is anything missing?

Answer 1

You need a lot more information to figure out what's going on here. Some possibilities include:

• Your "hard" examples are actually unlearnable. You can check this by seeing if you can at minimum overfit the hard examples, and if you as a human can label the hard examples correctly.

• Your network is not large enough to learn the hard examples. That is, you are sure that the hard examples can be learned, it's just that the network isn't sufficiently complex to do it. Again you can check this by seeing if you as a human are able to figure out the correct labels for the hard instances.

• Your curriculum learning may actually be hurting be the training. At the end of training, your network is only seeing examples that it has the most trouble with. These examples will cause the largest changes in the network, and you might have short-term gradient explosion. You can check if this is happening by seeing if you are getting significantly worse performance on "easy" examples at the end of training on the hard examples. One solution here might be something like slowly expanding the training set after you sort by difficulty. So if you have 10k easy examples and 10k hard examples, train on the 10k easy + 1k hard until you're satisified with the network performance, then grow the training set by adding the next 1k hardest, etc.