An issue for sub-word tokenization preprocessing transformer

Question

I'm stacked with executing the sub-word tokenization preprocessing to use transformer.

According to the tutorial on the article, I have executed the sample code.

However, one function was not defined properly and no hint to fix it on the article.

If you have any ideas to fix the code, could you help me?

Error

     22  return X, y
     23 
---> 24 X_train, y_train = preprocess(train_samples)
     25 X_val, y_val = preprocess(val_samples)

     12 def preprocess(samples):
     13  tag_index = {tag: i for i, tag in enumerate(schema)}
---> 14  tokenized_samples = list(tqdm(map(tokenize_sample, samples)))
     15  max_len = max(map(len, tokenized_samples))
     16  X = np.zeros((len(samples), max_len), dtype=np.int32)

TypeError: 'module' object is not callable

Code

This code is from the article to build a model for named entity recognition using transformer.

import numpy as np
import tqdm
 
def tokenize_sample(sample):
  seq = [
         (subtoken, tag)
         for token, tag in sample
         for subtoken in tokenizer(token)['input_ids'][1:-1]
         ]
  return [(3, 'O')] + seq + [(4, 'O')]

def preprocess(samples):
  tag_index = {tag: i for i, tag in enumerate(schema)}
  tokenized_samples = list(tqdm(map(tokenize_sample, samples)))
  max_len = max(map(len, tokenized_samples))
  X = np.zeros((len(samples), max_len), dtype=np.int32)
  y = np.zeros((len(samples), max_len), dtype=np.int32)
  for i, sentence in enumerate(tokenized_samples):
    for j, (subtoken_id, tag) in enumerate(sentence):
      X[i, j] = subtoken_id
      y[i,j] = tag_index[tag]
  return X, y

X_train, y_train = preprocess(train_samples)
X_val, y_val = preprocess(val_samples)

What I tried

I checked that the function, tokenize_sample is executable with the below code.

However, I'm not sure how to insert it to the original code.

for sample in samples:
  print(tokenize_sample(sample))

Answer 1

Your import of tqdm is incorrect. In your code you are trying to use tqdm as a function (note the parentheses) whereas you only imported the top level tqdm module, resulting the the error saying that the 'module' object is not callable (i.e. cannot be used as a function). Changing your import to import tqdm from tqdm imports the tqdm function and should solve error.