I am just playing with bert (Bidirectional Encoder Representation from Transformer)
Research Paper

Suppose I want to add any other model or layers like Convolutional Neural Network layers (CNN), Non Linear (NL) layers on top of BERT model. How can I do this?

I am not able to figure out where should I change in code of BERT. I am using the pytorch implementation of bert from huggingface.

This is what I want to do: enter image description here enter image description here

Please show steps to implement this using sudo code which will help me in implemention of cnn on top of BERT.


2 Answers 2



import transformers
import torch

tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased')
bert_model = transformers.BertModel.from_pretrained('bert-base-uncased')

max_seq = 100

def tokenize_text(df, max_seq):
    return [
        tokenizer.encode(text)[:max_seq] for text in df['text']

def pad_text(tokenized_text, max_seq):
    return np.array([el + [0] * (max_seq - len(el)) for el in tokenized_text])

def tokenize_and_pad_text(df, max_seq):
    tokenized_text = tokenize_text(df, max_seq)
    padded_text = pad_text(tokenized_text, max_seq)
    return torch.tensor(padded_text)

def targets_to_tensor(df):
    return torch.tensor(df['label'].values, dtype=torch.float32)

train_indices = tokenize_and_pad_text(small_train, max_seq)
val_indices = tokenize_and_pad_text(small_valid, max_seq)
test_indices = tokenize_and_pad_text(small_test, max_seq)

with torch.no_grad():
    x_train = bert_model(train_indices)[0]  
    x_val = bert_model(val_indices)[0]
    x_test = bert_model(test_indices)[0]

y_train = targets_to_tensor(small_train)
y_val = targets_to_tensor(small_valid)
y_test = targets_to_tensor(small_test)


import time
import torch.nn as nn
import torch.nn.functional as F
from sklearn.metrics import roc_auc_score
from torch.autograd import Variable

class KimCNN(nn.Module):
    def __init__(self, embed_num, embed_dim, class_num, kernel_num, kernel_sizes, dropout, static):
        super(KimCNN, self).__init__()
        V = embed_num
        D = embed_dim
        C = class_num
        Co = kernel_num
        Ks = kernel_sizes

        self.static = static
        self.embed = nn.Embedding(V, D)
        self.convs1 = nn.ModuleList([nn.Conv2d(1, Co, (K, D)) for K in Ks])
        self.dropout = nn.Dropout(dropout)
        self.fc1 = nn.Linear(len(Ks) * Co, C)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        if self.static:
            x = Variable(x)
        x = x.unsqueeze(1)  # (N, Ci, W, D)
        x = [F.relu(conv(x)).squeeze(3) for conv in self.convs1]  # [(N, Co, W), ...]*len(Ks)
        x = [F.max_pool1d(i, i.size(2)).squeeze(2) for i in x]  # [(N, Co), ...]*len(Ks)
        x = torch.cat(x, 1)
        x = self.dropout(x)  # (N, len(Ks)*Co)
        logit = self.fc1(x)  # (N, C)
        output = self.sigmoid(logit)
        return output

n_epochs = 50
batch_size = 10
lr = 0.01
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
loss_fn = nn.BCELoss()

def generate_batch_data(x, y, batch_size):
    i, batch = 0, 0
    for batch, i in enumerate(range(0, len(x) - batch_size, batch_size), 1):
        x_batch = x[i : i + batch_size]
        y_batch = y[i : i + batch_size]
        yield x_batch, y_batch, batch
    if i + batch_size < len(x):
        yield x[i + batch_size :], y[i + batch_size :], batch + 1
    if batch == 0:
        yield x, y, 1
train_losses, val_losses = [], []

for epoch in range(n_epochs):
    start_time = time.time()
    train_loss = 0

    for x_batch, y_batch, batch in generate_batch_data(x_train, y_train, batch_size):
        y_pred = model(x_batch)
        y_batch = y_batch.unsqueeze(1)
        loss = loss_fn(y_pred, y_batch)
        train_loss += loss.item()

    train_loss /= batch
    elapsed = time.time() - start_time

    model.eval() # disable dropout for deterministic output
    with torch.no_grad(): # deactivate autograd engine to reduce memory usage and speed up computations
        val_loss, batch = 0, 1
        for x_batch, y_batch, batch in generate_batch_data(x_val, y_val, batch_size):
            y_pred = model(x_batch)
            y_batch = y_batch.unsqueeze(1)
            loss = loss_fn(y_pred, y_batch)
            val_loss += loss.item()
        val_loss /= batch

        "Epoch %d Train loss: %.2f. Validation loss: %.2f. Elapsed time: %.2fs."
        % (epoch + 1, train_losses[-1], val_losses[-1], elapsed)

This blog explains well how to use CNN with BERT in Pytorch.


You could use HuggingFace's BertModel (transformers) as the base layer for your model and just like how you would build a neural network in Pytorch, you can build on top of it. HuggingFace's other BertModels are built in the same way. For reference you can take a look at their TokenClassification code over here. This hasn't been mentioned in the documentation much and I think it should.

For Tensorflow however, you would have convert the Bert Model into a Keras layer. I haven't really gone through it much or tried it, but I think this blog post does a pretty good job of explaining it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.