3
$\begingroup$

Tensorflow has built-in implementations for both, the Connectionist Temporal Classification (CTC) loss and a special seq2seq loss (weighted cross-entropy).

Since CTC loss is also intended to deal with seq2seq mappings, I wonder about how the two loss functions compare. I went through an excellent explanation of CTC loss, finding out that the target sequence is restricted not to be longer than the input sequence whereas this restriction does not exist for seq2seq loss.

Some questions I have (ranked by relevance):

  1. Are bidirectional LSTMs suited for seq2seq loss? I have only found b-LSTM implementations with CTC or even simpler loss functions. There is a post from 2016 saying that this was not possible at that time. Any updates?
  2. Are there conceptual differences apart from the many-to-one restriction in CTC loss that seq2seq does not have? E.g. does the seq2seq loss make a similar conditional independence assumption like the CTC loss?
  3. Do the loss functions have preferences regarding the optimizer (RMSProp, ADAM, Momentum etc.)
  4. Do the loss functions work better for particular sets of tasks?

Theoretical arguments and practical experiences are both appreciated.

$\endgroup$
  • $\begingroup$ Still interested. Happy about any thought :) $\endgroup$ – dopexxx Apr 8 '18 at 19:57
1
$\begingroup$

From my own work, I give a few, sparse empirical reports:

  1. bLSTMs are suitable for seq2seq loss in Tensorflow in case they are used within the encoder of an encoding-decoding scheme. I have not tried to use bLSTMs within the decoder.
  2. --
  3. seq2seq loss could be trained almost equally well with both RMSProp and ADAM on both G2P and P2G tasks.
  4. --
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.