I recently came across a paper on using (rather simple version of) LSTM for sentiment classification, and it describes its network settings as:

We randomize the parameters with uniform distribution U(-0.003, 0.003), set the clipping threshold of softmax layer as 200 and set learning rate as 0.01.

I am trying to reproduce their results with my code written in Tensorflow. I have to say I find it a bit confusing by the meaning of "set the clipping threshold of softmax layer as 200". 200???

Can someone explain this to me please so I know how to implement this using Tensorflow? Thanks so much!!!

Update, ok so I found the their code and below is how they use this clipping threshold of softmax layer:

for(int k = 0; k < softmax.outputG.length; k++)
    softmax.outputG[k] = 0.0;
softmax.outputG[goldPol] = 1.0 / softmax.output[goldPol];

// if ||g|| >= threshold, then g <- g * threshold / ||g|| 
if(Math.abs(softmax.outputG[goldPol]) > clippingThreshold)
    if(softmax.outputG[goldPol] > 0)
        softmax.outputG[goldPol] =  clippingThreshold;
        softmax.outputG[goldPol] =  -1.0 * clippingThreshold;

I still can't say I fully understand the use of this threshold..

  • $\begingroup$ You probably reading this one. One thing I can think of – they mean softmax with temprerature. Although they give the definition of softmax they are using which is without temperature parameter... Confusing! $\endgroup$ – Mikhail Yurasov Dec 23 '16 at 0:01
  • 1
    $\begingroup$ I updated my original post with some of their code that I found btw.. $\endgroup$ – Blue482 Dec 23 '16 at 14:00

I think what they were doing is just gradient clipping. It keeps the gradient of softmax layer between [-200,200]


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.