0
$\begingroup$

Background

I'm reading this article about a natural language task, named entity recognition and trying to load a pre-trained BERT model on Google colaboratory.

How can I fix an error to load a pre-trained BERT model?

Code

from transformers import AutoConfig, TFAutoModelForTokenClassification
MODEL_NAME = 'bert-base-german-cased' 
config = AutoConfig.from_pretrained(MODEL_NAME, num_labels=len(schema))
model = TFAutoModelForTokenClassification.from_pretrained(MODEL_NAME, config=config)
model.summary()

Error

I can understand that schema is not defined before the line, but I cannot find a clew on the article to fix it.

      1 from transformers import AutoConfig, TFAutoModelForTokenClassification
      2 MODEL_NAME = 'bert-base-german-cased'
----> 3 config = AutoConfig.from_pretrained(MODEL_NAME, num_labels=len(schema))
      4 model = TFAutoModelForTokenClassification.from_pretrained(MODEL_NAME, config=config)
      5 model.summary()

NameError: name 'schema' is not defined

What I tried

I checked previous blogpost following the advice from a comment, and found one description.

However, I'm not sure where to insert it to the original code.

For simplicity, we’ll truncate the sentences to a maximum length and pad shorter input sequences. But first, let us determine the set of all tags in the data and add an extra tag for the padding:

#code
schema = ['_'] + sorted({tag for sentence in samples for _, tag in sentence})

Is it correct understanding?

def load_data(filename: str):
   with open(filename, 'r') as file:
     lines = [line[:-1].split() for line in file]
     samples, start = [], 0
     for end, parts in enumerate(lines):
       if not parts:
         sample = [(token, tag.split('-')[-1]) for token, tag in lines[start:end]]
         samples.append(sample)
         start = end + 1
     if start < end:
       samples.append(lines[start:end])
     
     return samples

samples = load_data('data/01_raw/bag.conll')
train_samples = load_data('data/01_raw/bag.conll')
val_samples = load_data('data/01_raw/bgh.conll')
all_samples = train_samples + val_samples

schema = ['_'] + sorted({tag for sentence in samples for _, tag in sentence})

I checked the output.

print(schema)
#result
['_', 'AN', 'EUN', 'GRT', 'GS', 'INN', 'LD', 'LDS', 'LIT', 'MRK', 'O', 'ORG', 'PER', 'RR', 'RS', 'ST', 'STR', 'UN', 'VO', 'VS', 'VT']
$\endgroup$
6
  • 1
    $\begingroup$ You have to define the number of classes yourself, see also what the value means in their previous blogpost. $\endgroup$
    – Oxbowerce
    Aug 4 at 13:50
  • $\begingroup$ @Oxbowerce Thank you for your advice! I've edited my question and is it a correct understanding? $\endgroup$
    – halt
    Aug 4 at 15:41
  • 1
    $\begingroup$ Putting the code there would give an error since the samples variable is not defined. Have a look at the original blogpost you linked as there is a code block that shows you exactly what the order of the different lines should be. $\endgroup$
    – Oxbowerce
    Aug 4 at 15:49
  • $\begingroup$ @Oxbowerce Thank you for your reply. Actually, the samples variable was not defined outside of the load_data function on the original code. $\endgroup$
    – halt
    Aug 4 at 23:16
  • 1
    $\begingroup$ It is though, on the first code block in section two you see samples is defined after the load_data function as follows: samples = train_samples + val_samples. $\endgroup$
    – Oxbowerce
    Aug 5 at 7:24
1
$\begingroup$

The number of classes is something you have to define yourself depending on the problem you're working with. In the blogpost you've linked you see that they refer to a variable called schema, which is defined in in the previous blogpost to the one you've linked as follows: schema = ['_'] + sorted({tag for sentence in samples for _, tag in sentence}). This also refers to a variable called samples, which is defined as samples = train_samples + val_samples. Combining these pieces of code the correct preprocessing pipeline would be as follows:

def load_data(filename: str):
    with open(filename, 'r') as file:
        lines = [line[:-1].split() for line in file]
    samples, start = [], 0
    for end, parts in enumerate(lines):
        if not parts:
            sample = [(token, tag.split('-')[-1]) 
                        for token, tag in lines[start:end]]
            samples.append(sample)
            start = end + 1
    if start < end:
        samples.append(lines[start:end])
    return samples

train_samples = load_data('data/01_raw/bag.conll')
val_samples = load_data('data/01_raw/bgh.conll')
samples = train_samples + val_samples
schema = ['_'] + sorted({tag for sentence in samples for _, tag in sentence})
$\endgroup$
1
  • $\begingroup$ @Oxbwerce Thank you for your answer. This is a related question, I'd appreciate if you could check this one in your spare time. $\endgroup$
    – halt
    Aug 5 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.