11
$\begingroup$

I would like to train my LSTM with a "synthetic gradients" Decoupled Neural Interface (DNI).

How to decide on the number of layers and neurons for my DNI? Searching for them by trial end error or what's worse - by Genetic algorithm which would seem to outweigh [sic] defeat the purpose of Synthetic Gradients.

And, if my DNI is an LSTM itself - it seems it would take even longer to determine its optimal structure

SG speed up the training speed, by allowing multiple forward passes (with immediate weight adjustments), since DNI will already predict the future gradient.

However, we will lose time "experiencing" a few hundreds of training sessions only to find a optimal structure of DNI with which it will predict gradient the best way.

By that time we could have already finished our training with an oldschool Backprop through Time.

Also, how should we avoid our DNI overfitting, how to monitor and ensure it's not happening?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Please stick to one question per post. If you have multiple questions you can post them separately. $\endgroup$
    – D.W.
    May 24, 2018 at 1:46
  • 1
    $\begingroup$ Thanks, I would argue these are acceptable to be in same question - number of neurons and overfitting are 2 sides of the same coin $\endgroup$
    – Kari
    May 24, 2018 at 5:15
  • 1
    $\begingroup$ I'm not persuaded. Many (perhaps even most?) of the techniques for dealing with overfitting have nothing to do with the number of neurons. $\endgroup$
    – D.W.
    May 24, 2018 at 5:21
  • 1
    $\begingroup$ from this post: stats.stackexchange.com/a/306607/187816 Increasing the number of hidden units and/or layers may lead to overfitting because it will make it easier for the neural network to memorize the training set, that is to learn a function that perfectly separates the training set but that does not generalize to unseen data. $\endgroup$
    – Kari
    May 24, 2018 at 5:31
  • 1
    $\begingroup$ As I understand, overfitting is occurring due to some weights "specializing" too much to specific tasks. If we have too many neurons there is just too many to choose from, they are all going to be "specialized" $\endgroup$
    – Kari
    May 24, 2018 at 15:17

1 Answer 1

Reset to default
1
$\begingroup$

Several months later, I have a couple of insights on it:

Also, how should we avoid our DNI overfitting, how to monitor and ensure it's not happening?

I don't think it matters, because DNI overfitting on the gradient is actually what we want. We want it to figure-out the pattern which reduces the error in the fastest way for our data. However, as always, we should pay attention to the Validation of our entire network while doing so.

That probably means the more DNI neurons the better, as long as the designated training data stays unmodified o_O
If new or extra data is used for training, we should throw away our DNI, and just train once more, to overfit them on this new "adjusted" training-data.

Also, more on Synthetic Grads here

wish for a better answer, so won't select this one - plz post if you have a better one

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.