5
$\begingroup$

I've found the script for punctuation restoration. And I have one question about this method.

I will briefly explain the logic of the author. One of four tokens is assigned for each word: Other (0), PERIOD (1), COMMA (2), QUESTION (3). Further, all words are converted to BERT tokens. Here is an example:

  2045  0
  2003  0
  2200  0
  2210  0
  3983  0
  2301  0
  2974  0
  1999  0
  2068  2

Next, we do padding. We set a segment (e.g. eight words) and for each word we take two words before a token and four words after a token. Also, we add a padding token after each word. For the very first word, there are no words before. Therefore, a word are taken from the end. Similarly, for the last word, there are no words after and therefore a word are taken from the beginning. Here is an example of it.

[1999, 2068, 2045, 0, 2003, 2200, 2210, 3983] 0
[2068, 2045, 2003, 0, 2200, 2210, 3983, 2301] 0
[2045, 2003, 2200, 0, 2210, 3983, 2301, 2974] 0
[2003, 2200, 2210, 0, 3983, 2301, 2974, 1999] 0
[2200, 2210, 3983, 0, 2301, 2974, 1999, 2068] 0
[2210, 3983, 2301, 0, 2974, 1999, 2068, 2045] 0
[3983, 2301, 2974, 0, 1999, 2068, 2045, 2003] 0
[2301, 2974, 1999, 0, 2068, 2045, 2003, 2200] 0
[2974, 1999, 2068, 0, 2045, 2003, 2200, 2210] 2

The first column contains tokens, and the second column contains punctuation symbols. In first column, "0" corresponds to a padding. Next we do TensorDataset, and than DataLoader. In the second column, '0' corresponds to "other", and '2' corresponds to a "period". Finaly we train a model:

for inputs, labels in data_loader_train:
       inputs, labels = inputs.cuda(), labels.cuda()
       output = model(inputs)

The algorithm works well, but I do not understand the following. What's the point of putting padding in the middle? Maybe we can do punctuation restoration with BERT in a more simple way?

$\endgroup$
1
  • $\begingroup$ Hi, I have a question regarding the same model. In the train.py there are 2 processes: "training top layers" and "training all layers". Do I need to run both of them or just one is enough? For me setting "p.requires_grad = False" will lead to an error: "RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn". Thanks a lot! $\endgroup$ – hoang tran Nov 26 '20 at 14:09
1
$\begingroup$

I am also confused by the method of BertPunc. Actually I can't believe it really works. Beacause the method use BertLanguageModel to do the job, and LanguseModel's main capabiltiy is to predict the next word of a word. So also use your example, BertPunc inputs 8 words, the output of bert should be 8 next words of the inputs. But BertPunc added a Linear layer to only get 1 output value, which is to predict punctuation.

But I think we should use bert the way like the graph below:enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.