My task is to generate keywords from sentences.

I pretrain a text-generation model. I mask the sentences' tokens and predict the whole sentences' tokens.

Pretraining batch_size = 8 and step = 1000000

I haven't observed improvement from pretraining. BLEU score is 10.5 for not pretraining, BLEU score is 9.5 for pretraining.


I take the python code from


hidden_size = 512 num_encoder_layers = 3 num_decoder_layers = 3


The task is to generate keyword from sentences. The keyword may not appear in the sentences. So input masked sentences to predict whole sentences, it is not benefit the keywords generation task. Input masked sentences to predict whole sentences, it do not have relation to the keywords generation task. Am I right? Is it the reason that pretraining do not improve the BLEU score?

Thank you very much.


1 Answer 1


1, I pad some zeros in the input tokens for multi sentences. The output positions of output tokens should be exactly same to the input tokens, which means I should keep the padding zeros in the output tokens.

2, The pretraining time should be longer.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.