0
$\begingroup$

As a learning project, I'm training a BERT model with the CoLA dataset to detect sentence acceptability. Unfortunately my model is learning to classify every instance as "acceptable", and I'm not sure what is going wrong with my code. Can anyone provide any help or insight into why this happens?

Technical details

I'm using hugging face's transformers library with PyTorch.

The (stripped version of the) code is as follows. Instrumentation and other details have been left out so the code is more readable.

tokenizer = transformers.AutoTokenizer.from_pretrained('bert-base-cased')

model = transformers.AutoModelForSequenceClassification.from_pretrained(
    'bert-base-cased',
    num_labels=2,
)

optimizer = transformers.AdamW(
    model.parameters(),
    lr=2e-5,
    eps=1e-8,
)

# The next lines read the CoLA dataset and split it for training and validation
training_dataloader = ...
validation_dataloader = ...

for epoch in range(4):
    train_loss = 0

    for batch in tqdm(train_dataloader):
        model.train()

        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        labels = batch['label'].to(device)

        model.zero_grad()

        model_output = model(
            input_ids=input_ids,
            attention_mask=attention_mask,
            labels=labels,
            return_dict=True,
        )

        batch_loss = model_output.loss.sum()

        train_loss += batch_loss.item()

        batch_loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()

        # HERE: Measure performance after learning from each batch here with validation_dataloader

I've checked and the code that reads the dataset seems correct, so it seems that the problem lies either in measuring the performance of the model or in the learning phase.

To measure the performance, I'm running the model over the validation split and then converting the logits into actual classifications as follows:

tp = fn = fp = tn = 0

for batch in validation_dataloader:
    input_ids = batch['input_ids'].to(device)
    attention_mask = batch['attention_mask'].to(device)
    labels = batch['label'].to(device)

    model(
        input_ids=input_ids,
        attention_mask=attention_mask,
        labels=labels,
        return_dict=True,
    )

    expected = batch['label']
    predictions = output.logits[:,1] > output.logits[:,0] # Is this correct?

    tp += sum(1 for exp, pred in zip(expected, predictions) if     exp and     pred)
    fn += sum(1 for exp, pred in zip(expected, predictions) if     exp and not pred)
    fp += sum(1 for exp, pred in zip(expected, predictions) if not exp and     pred)
    tn += sum(1 for exp, pred in zip(expected, predictions) if not exp and not pred)

Is the line predictions = ... correct?

$\endgroup$
6
  • 1
    $\begingroup$ is the dataset balanced? $\endgroup$ – Bruno Lubascher Dec 10 '20 at 18:50
  • $\begingroup$ It's 70/30, so not really unbalanced. I've also tried to undersample my dataset to ensure 50/50 distribution of labels but the problem persists. $\endgroup$ – jdferreira Dec 10 '20 at 22:54
  • $\begingroup$ I've added a bit more on how I'm measuring validation, since it may also be the point of failure. $\endgroup$ – jdferreira Dec 10 '20 at 22:59
  • $\begingroup$ predictions = torch.argmax(output.logits,dim=1) should do the job $\endgroup$ – Ashwin Geet D'Sa Dec 10 '20 at 23:42
  • $\begingroup$ That may be a better way to express the computation, but computes the exact same thing, I think... $\endgroup$ – jdferreira Dec 11 '20 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.