Error to load a pre-trained BERT model

Question

Background

I'm reading this article about a natural language task, named entity recognition and trying to load a pre-trained BERT model on Google colaboratory.

How can I fix an error to load a pre-trained BERT model?

Code

from transformers import AutoConfig, TFAutoModelForTokenClassification
MODEL_NAME = 'bert-base-german-cased' 
config = AutoConfig.from_pretrained(MODEL_NAME, num_labels=len(schema))
model = TFAutoModelForTokenClassification.from_pretrained(MODEL_NAME, config=config)
model.summary()

Error

I can understand that schema is not defined before the line, but I cannot find a clew on the article to fix it.

      1 from transformers import AutoConfig, TFAutoModelForTokenClassification
      2 MODEL_NAME = 'bert-base-german-cased'
----> 3 config = AutoConfig.from_pretrained(MODEL_NAME, num_labels=len(schema))
      4 model = TFAutoModelForTokenClassification.from_pretrained(MODEL_NAME, config=config)
      5 model.summary()

NameError: name 'schema' is not defined

What I tried

I checked previous blogpost following the advice from a comment, and found one description.

However, I'm not sure where to insert it to the original code.

For simplicity, we’ll truncate the sentences to a maximum length and pad shorter input sequences. But first, let us determine the set of all tags in the data and add an extra tag for the padding:

#code
schema = ['_'] + sorted({tag for sentence in samples for _, tag in sentence})

Is it correct understanding?

def load_data(filename: str):
   with open(filename, 'r') as file:
     lines = [line[:-1].split() for line in file]
     samples, start = [], 0
     for end, parts in enumerate(lines):
       if not parts:
         sample = [(token, tag.split('-')[-1]) for token, tag in lines[start:end]]
         samples.append(sample)
         start = end + 1
     if start < end:
       samples.append(lines[start:end])
     
     return samples

samples = load_data('data/01_raw/bag.conll')
train_samples = load_data('data/01_raw/bag.conll')
val_samples = load_data('data/01_raw/bgh.conll')
all_samples = train_samples + val_samples

schema = ['_'] + sorted({tag for sentence in samples for _, tag in sentence})

I checked the output.

print(schema)
#result
['_', 'AN', 'EUN', 'GRT', 'GS', 'INN', 'LD', 'LDS', 'LIT', 'MRK', 'O', 'ORG', 'PER', 'RR', 'RS', 'ST', 'STR', 'UN', 'VO', 'VS', 'VT']

Answer 1

The number of classes is something you have to define yourself depending on the problem you're working with. In the blogpost you've linked you see that they refer to a variable called schema, which is defined in in the previous blogpost to the one you've linked as follows: schema = ['_'] + sorted({tag for sentence in samples for _, tag in sentence}). This also refers to a variable called samples, which is defined as samples = train_samples + val_samples. Combining these pieces of code the correct preprocessing pipeline would be as follows:

def load_data(filename: str):
    with open(filename, 'r') as file:
        lines = [line[:-1].split() for line in file]
    samples, start = [], 0
    for end, parts in enumerate(lines):
        if not parts:
            sample = [(token, tag.split('-')[-1]) 
                        for token, tag in lines[start:end]]
            samples.append(sample)
            start = end + 1
    if start < end:
        samples.append(lines[start:end])
    return samples

train_samples = load_data('data/01_raw/bag.conll')
val_samples = load_data('data/01_raw/bgh.conll')
samples = train_samples + val_samples
schema = ['_'] + sorted({tag for sentence in samples for _, tag in sentence})