# UniLM - Unified Language Model for summarization

The UniLM claims to be the best approach for summarization task. But there doesn't seem to be any tutorial or how-to section in the README.md or any other blog. How exactly can I use this state-of-the-art library for abstractive summary generation?

Paper

P.S. A newbie in NLP. Sorry if this is a dumb question.

• @MathiasMüller I have edited my question to include the requested links. Feb 16 '20 at 11:19
• The Github repo README does contain instructions to train/run the model. What did you find lacking / unclear about those instructions? Feb 16 '20 at 11:53
• @MathiasMüller Where exactly are you pointing to? Is it the evaluation part? I do not see anything that guides to summary generation. Thank you for having a look. Feb 16 '20 at 11:54
• github.com/microsoft/… Feb 16 '20 at 11:56

Here's what you should do

1. Prepare your dataset: Follow similar instructions as described in the paper and preprocess your dataset. This will be your major task as after this you will only have to fine-tune the model. If you don't have a dataset, you can use the dataset used in this research paper, which can be downloaded from here.

2. Download the pre-trained model. Or you can choose to start from the provided fine-tuned model checkpoint (from the link). You will have to check which version of the model works best for your dataset. If you select model fine-tuned for summarization task and your dataset is similar to CNN/DailyMail dataset [37] and Gigaword [36] you can skip fine-tuning.

3. Fine-tune model: In this step, you will be using the command mentioned in the readme of the Github repository. Note that, there are some parameters that should be according to the language model you have downloaded in the previous step. Based on the size of your dataset you can change the number of epochs in the following command. You should also note that this will require GPU. The repository readme recommends 2 or 4 v100-32G GPU cards for finetuning the model.

    OUTPUT_DIR=/{path_of_fine-tuned_model}/
MODEL_RECOVER_PATH=/{path_of_pre-trained_model}/unilmv1-large-cased.bin
export PYTORCH_PRETRAINED_BERT_CACHE=/{tmp_folder}/bert-cased-pretrained-cache
export CUDA_VISIBLE_DEVICES=0,1,2,3
python biunilm/run_seq2seq.py --do_train --fp16 --amp --num_workers 0 \
--bert_model bert-large-cased --new_segment_ids --tokenized_input \
--data_dir ${DATA_DIR} \ --output_dir${OUTPUT_DIR}/bert_save \
--log_dir ${OUTPUT_DIR}/bert_log \ --model_recover_path${MODEL_RECOVER_PATH} \
--max_seq_length 192 --max_position_embeddings 192 \
--trunc_seg a --always_truncate_tail --max_len_a 0 --max_len_b 64 \
--learning_rate 0.00003 --warmup_proportion 0.1 --label_smoothing 0.1 \
--num_train_epochs 30

1. Evaluate your model: Use biunilm/decode_seq2seq.py to decode (predict the output of evaluation dataset) and use the provided evaluation script to evaluate the trained model.

2. Use the trained model: In order to use this model to make a prediction, you can simply write your own python code to:

• load Pytorch pre-trained model using pytorch_pretrained_bert library as used in decode_seq2seq.py file
• predict the output and detokenize the output.

Here is the logic which you can use:

model = BertForSeq2SeqDecoder.from_pretrained(long_list_of_arguments)

Note that this is not the complete logic. This code just show how github repository code has handled the saved model and used it to make predictions. Use traces to convert ids to tokens and detokenize the output tokens (as used in the code here). The detokenization step is necessary as the input sequence is tokenized to subword units by WordPiece.