0
$\begingroup$

For example, let's tokenize a sentece "why isn't Alex' text tokenizing":

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
tokens = tokenizer.tokenize("why isn't Alex' text tokenizing")

We are getting the next output:

['why', 'isn', "'", 't', 'Alex', "'", 'text', 'token', '##izing']

I want to convert it to:

why isn't Alex' text tokenizing

It seems the tokenizer doesn't do its job in the best way. If after tokenization we would have something like that:

['why', 'isn', "##'", '##t', 'Alex', "##'", 'text', 'token', '##izing']

It would be easy to convert:

tokens = ['why', 'isn', "##'", '##t', 'Alex', "##'", 'text', 'token', '##izing']
restored_text = [None] * len(tokens)
tokens.extend(['#', '#'])
for i in range(len(tokens )-2):
    if re.findall("#{2}", tokens[i+1]):
      restored_text [i] = tokens[i] + tokens[i+1].replace('##', '')
      if re.findall("#{2}", tokens [i+2]):
        restored_text[i] = restored_text[i] + tokens[i+2].replace('##', '')
    else:
      restored_text[i] = tokens[i]

restored_text_without_masks = []
for i in range(len(restored_text)):
    if not restored_text[i].startswith('#'):
        restored_text_without_masks.append(restored_text[i])
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.