3
$\begingroup$

I recently came across a paper on using (rather simple version of) LSTM for sentiment classification, and it describes its network settings as:

We randomize the parameters with uniform distribution U(-0.003, 0.003), set the clipping threshold of softmax layer as 200 and set learning rate as 0.01.

I am trying to reproduce their results with my code written in Tensorflow. I have to say I find it a bit confusing by the meaning of "set the clipping threshold of softmax layer as 200". 200???

Can someone explain this to me please so I know how to implement this using Tensorflow? Thanks so much!!!

Update, ok so I found the their code and below is how they use this clipping threshold of softmax layer:

for(int k = 0; k < softmax.outputG.length; k++)
    softmax.outputG[k] = 0.0;
softmax.outputG[goldPol] = 1.0 / softmax.output[goldPol];

// if ||g|| >= threshold, then g <- g * threshold / ||g|| 
if(Math.abs(softmax.outputG[goldPol]) > clippingThreshold)
{
    if(softmax.outputG[goldPol] > 0)
        softmax.outputG[goldPol] =  clippingThreshold;
    else
        softmax.outputG[goldPol] =  -1.0 * clippingThreshold;
}

I still can't say I fully understand the use of this threshold..

$\endgroup$
2
  • $\begingroup$ You probably reading this one. One thing I can think of – they mean softmax with temprerature. Although they give the definition of softmax they are using which is without temperature parameter... Confusing! $\endgroup$ Commented Dec 23, 2016 at 0:01
  • 1
    $\begingroup$ I updated my original post with some of their code that I found btw.. $\endgroup$
    – Blue482
    Commented Dec 23, 2016 at 14:00

1 Answer 1

4
$\begingroup$

I think what they were doing is just gradient clipping. It keeps the gradient of softmax layer between [-200,200]

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.