7
$\begingroup$

First of all we have a classification task.
So we use the typical softmax cross entropy to classify.

Current implementation of curriculum learning is as follows.

  • First we train our best version of the neural net
  • At the last epoch we get all of the cross entropies for all the instances. So smaller cross entropies are better classified and larger ones not so well.
  • Then we sort the instances based on the cross entropies.
  • Then we start training the instances from easier to harder as the curriculum learning theory suggests

Note that we have already experimented with various steps and repetitions. So in one example we took the first 200 batches and trained them two times before going to the next batch and so on until an epoch is completed.
In another example we took the first 10 batches and trained them only once and then the next 10 and the next 10 and so on until the end of the epoch.

All of the experiments so far have concluded that the neural network is has a relatively ok accuracy at the beginning and this gets worsen as the more difficult instances come along. The final accuracy is much worse than expected and in addition the maximum accuracy is still quite bad.

Why is this curriculum learning not working? Is anything missing?

$\endgroup$
  • 1
    $\begingroup$ In my experience trying similar things for Kaggle competitions, this approach does not work well at all with neural networks. I think partially this is because the curriculum is being defined by what is easy/hard to fit by the network, as opposed to easy/hard to deal with conceptually. So as well as the real difficult items, you save up the worst noisy/incorrect data to feed in at the end. However, I don't know nearly enough about this subject to put forward an answer . . . $\endgroup$ – Neil Slater Feb 28 '17 at 16:36
  • $\begingroup$ Thanks for your response. So you are specifying neural networks. Have you seen curriculum learning work in other cases? $\endgroup$ – George Pligoropoulos Mar 1 '17 at 19:57
  • 1
    $\begingroup$ No I have not. I have basically tried the same approach as you outline in the question and got similar results, a couple of times. $\endgroup$ – Neil Slater Mar 1 '17 at 21:54
5
$\begingroup$

You need a lot more information to figure out what's going on here. Some possibilities include:

  • Your "hard" examples are actually unlearnable. You can check this by seeing if you can at minimum overfit the hard examples, and if you as a human can label the hard examples correctly.

  • Your network is not large enough to learn the hard examples. That is, you are sure that the hard examples can be learned, it's just that the network isn't sufficiently complex to do it. Again you can check this by seeing if you as a human are able to figure out the correct labels for the hard instances.

  • Your curriculum learning may actually be hurting be the training. At the end of training, your network is only seeing examples that it has the most trouble with. These examples will cause the largest changes in the network, and you might have short-term gradient explosion. You can check if this is happening by seeing if you are getting significantly worse performance on "easy" examples at the end of training on the hard examples. One solution here might be something like slowly expanding the training set after you sort by difficulty. So if you have 10k easy examples and 10k hard examples, train on the 10k easy + 1k hard until you're satisified with the network performance, then grow the training set by adding the next 1k hardest, etc.

| improve this answer | |
$\endgroup$
  • $\begingroup$ We tried to bring the curriculum step at a low level but still not any remarkable results. Following this approach the new goal is to optimize the neural network at every curriculum level to achieve best results. This would required tweaking of the hyperparameters (L2 regularization factor and Dropout Keep probabilities) at every curriculum level! We do not know any ways of doing this efficiently / automatically.. $\endgroup$ – George Pligoropoulos Mar 6 '17 at 1:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.