# BertPunc (punctuation restoration with BERT)

I've found the script for punctuation restoration. And I have one question about this method.

I will briefly explain the logic of the author. One of four tokens is assigned for each word: Other (0), PERIOD (1), COMMA (2), QUESTION (3). Further, all words are converted to BERT tokens. Here is an example:

  2045  0
2003  0
2200  0
2210  0
3983  0
2301  0
2974  0
1999  0
2068  2


Next, we do padding. We set a segment (e.g. eight words) and for each word we take two words before a token and four words after a token. Also, we add a padding token after each word. For the very first word, there are no words before. Therefore, a word are taken from the end. Similarly, for the last word, there are no words after and therefore a word are taken from the beginning. Here is an example of it.

[1999, 2068, 2045, 0, 2003, 2200, 2210, 3983] 0
[2068, 2045, 2003, 0, 2200, 2210, 3983, 2301] 0
[2045, 2003, 2200, 0, 2210, 3983, 2301, 2974] 0
[2003, 2200, 2210, 0, 3983, 2301, 2974, 1999] 0
[2200, 2210, 3983, 0, 2301, 2974, 1999, 2068] 0
[2210, 3983, 2301, 0, 2974, 1999, 2068, 2045] 0
[3983, 2301, 2974, 0, 1999, 2068, 2045, 2003] 0
[2301, 2974, 1999, 0, 2068, 2045, 2003, 2200] 0
[2974, 1999, 2068, 0, 2045, 2003, 2200, 2210] 2


The first column contains tokens, and the second column contains punctuation symbols. In first column, "0" corresponds to a padding. Next we do TensorDataset, and than DataLoader. In the second column, '0' corresponds to "other", and '2' corresponds to a "period". Finaly we train a model:

for inputs, labels in data_loader_train:
inputs, labels = inputs.cuda(), labels.cuda()
output = model(inputs)


The algorithm works well, but I do not understand the following. What's the point of putting padding in the middle? Maybe we can do punctuation restoration with BERT in a more simple way?

• Hi, I have a question regarding the same model. In the train.py there are 2 processes: "training top layers" and "training all layers". Do I need to run both of them or just one is enough? For me setting "p.requires_grad = False" will lead to an error: "RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn". Thanks a lot! – hoang tran Nov 26 '20 at 14:09

## 1 Answer

I am also confused by the method of BertPunc. Actually I can't believe it really works. Beacause the method use BertLanguageModel to do the job, and LanguseModel's main capabiltiy is to predict the next word of a word. So also use your example, BertPunc inputs 8 words, the output of bert should be 8 next words of the inputs. But BertPunc added a Linear layer to only get 1 output value, which is to predict punctuation.

But I think we should use bert the way like the graph below: