I'm stacked with executing the sub-word tokenization preprocessing to use transformer.

According to the tutorial on the article, I have executed the sample code.

However, one function was not defined properly and no hint to fix it on the article.

If you have any ideas to fix the code, could you help me?


     22  return X, y
---> 24 X_train, y_train = preprocess(train_samples)
     25 X_val, y_val = preprocess(val_samples)

     12 def preprocess(samples):
     13  tag_index = {tag: i for i, tag in enumerate(schema)}
---> 14  tokenized_samples = list(tqdm(map(tokenize_sample, samples)))
     15  max_len = max(map(len, tokenized_samples))
     16  X = np.zeros((len(samples), max_len), dtype=np.int32)

TypeError: 'module' object is not callable


This code is from the article to build a model for named entity recognition using transformer.

import numpy as np
import tqdm
def tokenize_sample(sample):
  seq = [
         (subtoken, tag)
         for token, tag in sample
         for subtoken in tokenizer(token)['input_ids'][1:-1]
  return [(3, 'O')] + seq + [(4, 'O')]

def preprocess(samples):
  tag_index = {tag: i for i, tag in enumerate(schema)}
  tokenized_samples = list(tqdm(map(tokenize_sample, samples)))
  max_len = max(map(len, tokenized_samples))
  X = np.zeros((len(samples), max_len), dtype=np.int32)
  y = np.zeros((len(samples), max_len), dtype=np.int32)
  for i, sentence in enumerate(tokenized_samples):
    for j, (subtoken_id, tag) in enumerate(sentence):
      X[i, j] = subtoken_id
      y[i,j] = tag_index[tag]
  return X, y

X_train, y_train = preprocess(train_samples)
X_val, y_val = preprocess(val_samples)

What I tried

I checked that the function, tokenize_sample is executable with the below code.

However, I'm not sure how to insert it to the original code.

for sample in samples:

This look like a tqdm problem. Both the module tqdm and the main function tqdm have the same name. This often create some problem as people will just:

import tqdm

When the right import is:

from tqdm import tqdm

See here: https://stackoverflow.com/questions/39323182/tqdm-module-object-is-not-callable

(The first step might be to just remove tqdm from the line you want to execute to test if this is the origin of the problem)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.