I am reading Character-Level Language Modeling with Deeper Self-Attention from Rami Al-Rfou. In the second page, they had mentioned about Auxiliary Losses which can speed-up the model convergence and as an additional regularizer. They said they had 3 kinds of auxiliary losses:

  1. Auxiliary losses at intermediate sequence positions
  2. Auxiliary losses from intermediate hidden representations
  3. Auxiliary losses at target positions multiple steps.

However, I cannot find any information or reference explaining auxiliary losses. I would like to know:


Auxiliary losses are additional terms added to your global loss to induce learning further upstream in your model and can be seen as a way to combat vanishing gradients.

An early reference would be how Google's Inception model uses auxiliary classifiers. The general idea is that you take an intermediate output of your model and use it as the prediction in a separate loss function. So during training, your model is acting as both a global model (all parameters subject to that final global loss) and a model composed of sub-models with specific terms applying exclusively to your auxiliary tasks.

In practical terms, you can follow this example. If the link ever dies, here's a generic view:

# let's say we have two auxiliary losses, and thus two auxiliary outputs
final_task_prediction, predictions = model.forward(x)

global_loss = loss(final_task_prediction, y)
intermediate_loss_1 = aux_loss_1(predictions[0], aux_y_1)
intermediate_loss_2 = aux_loss_2(predictions[1], aux_y_2)

# can weight auxiliary losses however we choose
total_loss = global_loss + .4*intermediate_loss_1 + .8*intermediate_loss_2

As an example, say we have an input sequence ABCDEF. To quickly run through the auxiliary losses in the Al-Rfou et al. paper

  • Multiple positions works as an auxiliary task of "given A, predict B, then given B (and our internal memory of A), predict C" (etc)

  • Intermediate hidden representations works the same as multiple positions, except intermediate layers also get their own loss terms

  • Multiple predictions breaks the target prediction sequence into multiple subsequences, so our auxiliary task is "given A predict B,C ; given B, predict C,D" (etc)

Hope this helps!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.