I want to use transformer xl for text classification tasks. But I don't know the architect model for the text classification task. I use dense layers with activation softmax for logits output from the transformer xl model, but this doesn't seem right. when training I see accuracy is very low.

Output of my model:

output, logits transformer-xl

My training step:

dense layer

New contributor
Dat Le is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • $\begingroup$ Classification with Transformer XL should be similar to classification with BERT, you must be doing something wrong. (But bidirectional models like BERT or RoBERTa might be a better choice because they are bidirectional.) Can you provide more details? What exactly do you use as input to the classifier? $\endgroup$
    – Jindřich
    May 12 at 8:36
  • $\begingroup$ @Jindřich I'm using IMDb dataset. and input is data was tokenizer $\endgroup$
    – Dat Le
    May 12 at 8:42


Your Answer

Dat Le is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.