I'm stacked with executing the sub-word tokenization preprocessing to use transformer.
According to the tutorial on the article, I have executed the sample code.
However, one function was not defined properly and no hint to fix it on the article.
If you have any ideas to fix the code, could you help me?
Error
22 return X, y
23
---> 24 X_train, y_train = preprocess(train_samples)
25 X_val, y_val = preprocess(val_samples)
12 def preprocess(samples):
13 tag_index = {tag: i for i, tag in enumerate(schema)}
---> 14 tokenized_samples = list(tqdm(map(tokenize_sample, samples)))
15 max_len = max(map(len, tokenized_samples))
16 X = np.zeros((len(samples), max_len), dtype=np.int32)
TypeError: 'module' object is not callable
Code
This code is from the article to build a model for named entity recognition using transformer.
import numpy as np
import tqdm
def tokenize_sample(sample):
seq = [
(subtoken, tag)
for token, tag in sample
for subtoken in tokenizer(token)['input_ids'][1:-1]
]
return [(3, 'O')] + seq + [(4, 'O')]
def preprocess(samples):
tag_index = {tag: i for i, tag in enumerate(schema)}
tokenized_samples = list(tqdm(map(tokenize_sample, samples)))
max_len = max(map(len, tokenized_samples))
X = np.zeros((len(samples), max_len), dtype=np.int32)
y = np.zeros((len(samples), max_len), dtype=np.int32)
for i, sentence in enumerate(tokenized_samples):
for j, (subtoken_id, tag) in enumerate(sentence):
X[i, j] = subtoken_id
y[i,j] = tag_index[tag]
return X, y
X_train, y_train = preprocess(train_samples)
X_val, y_val = preprocess(val_samples)
What I tried
I checked that the function, tokenize_sample is executable with the below code.
However, I'm not sure how to insert it to the original code.
for sample in samples:
print(tokenize_sample(sample))