I have been doing a thesis in my citation classifications. I just implemented Bert model for the classification of citations. I have 4 output classes and I give an input sentence and my model returns an output that tells the category of citation. Now my supervisor gave me another task.

You have to search that whether it is possible to extend BERT or any transformer model using manual features. e.g. You are currently giving a sentence as the only input followed by its class. What if you can give a sentence, and some other features as input; as we do in other classifiers?

I need some guidance about this problem. How can I add an extra feature in my Bert model and the feature would be categorical not numerical.

This is my code what I have done for implementation of BERT model I want to add manual features in my code

I am using Bert Tokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

Then I am making dataset of inputs ids and attention masks of size 256

X_input_ids = np.zeros((len(df), 256))
X_attn_masks = np.zeros((len(df), 256))

This is my function of tokenization of sentences

def generate_training_data(df, ids, masks, tokenizer):
    for i, text in tqdm(enumerate(df['Citing Sentence'])):
        tokenized_text = tokenizer.encode_plus(
        ids[i, :] = tokenized_text.input_ids
        masks[i, :] = tokenized_text.attention_mask
    return ids, masks

Then I am generating input_ids and attention_masks

from tqdm.auto import tqdm
X_input_ids, X_attn_masks = generate_training_data(df, X_input_ids, X_attn_masks, tokenizer)

Then I made set of size 4 because my output has 4 categories

  1. Related work
  2. Comparison
  3. Using the work
  4. Extending the work

One-hot encoded target tensor. Here follow-up is my output array array

labels = np.zeros((len(df), 4))
labels[np.arange(len(df)), df['Follow-up'].values] = 1
dataset = tf.data.Dataset.from_tensor_slices((X_input_ids, X_attn_masks, labels))

def CitationDatasetMapFunction(input_ids, attn_masks, labels):
    return {
        'input_ids': input_ids,
        'attention_mask': attn_masks
    }, labels

converting to required format for tensorflow dataset

dataset = dataset.map(CitationDatasetMapFunction)

batch size, drop any left out tensor

dataset = dataset.shuffle(10000).batch(16, drop_remainder=True)

for each 4 batch of data we will have len(df)//16 samples, take 80% of that for train.

p = 0.8
train_size = int((len(df)/16)*p) 

train_dataset = dataset.take(train_size)
val_dataset = dataset.skip(train_size)

Summarising the model

from transformers import TFBertModel
model = TFBertModel.from_pretrained('bert-base-cased') # bert base model with pretrained weights

input_ids = tf.keras.layers.Input(shape=(256,), name='input_ids', dtype='int32')
attn_masks = tf.keras.layers.Input(shape=(256,), name='attention_mask', dtype='int32')

bert_embds = model(input_ids, attention_mask=attn_masks)[1] # 0 -> activation layer (3D), 1 -> pooled output layer (2D)
intermediate_layer = tf.keras.layers.Dense(512, activation='relu', name='intermediate_layer')(bert_embds)
output_layer = tf.keras.layers.Dense(4, activation='softmax', name='output_layer')(intermediate_layer) # softmax -> calcs probs of classes

citation_model = tf.keras.Model(inputs=[input_ids, attn_masks], outputs=output_layer)

2 Answers 2


Standard BERT models take 768 (1024) dimensional vectors as their input. There is an encoding step that tokenizes and encodes a sentence from a string to a 768-dimensional vector. You can make changes in your BERT model or Tokenizer.

Change in BERT model: You can add some extra dimension to your BERT model to take more than a 768-dimensional vector. So, you will be now providing a 768 vector of your sentences and some additional features as input in your BERT model. Making this change will require writing a custom BERT model and can be a bit difficult for beginners.

Change in Tokenizer model: On the other hand, you can train a custom tokenizer for your BERT model which will output a vector with less than 768 dimensions and you can use the leftover dimension as your categorical feature. Hence, you will input a 768 vector in your BERT model. This does not require you to change the BERT model and can be a bit easier.

Disclaimer This change will not necessarily make the model perform better on your task, it might even make the performance worse. You have to apply trial and error to find the methods that suit best for your task.


welcome to the Datascience SE community.

To answer the OP, one potential approach could be to combine the categorical feature as a text with the given text input/sentence. ex. "Citation text" + "categorical feature" text.

The following article elaborates more on the same approach. https://mccormickml.com/2021/06/29/combining-categorical-numerical-features-with-bert

  • $\begingroup$ tokenized_text = tokenizer.encode_plus( text, max_length=256, truncation=True, padding='max_length', add_special_tokens=True, return_tensors='tf' ) this is the function of tokenization $\endgroup$ Sep 2, 2022 at 9:44
  • $\begingroup$ In tokenization function I am not using any additional features I am just passing a simple sentence $\endgroup$ Sep 2, 2022 at 9:46
  • $\begingroup$ I am stuck with work, as of now. Meanwhile, I would recommend you to give this a go mccormickml.com/2021/06/29/…, as this approach might help you, Atleast in the data preparation step. I found the solution, pretty much tailored to your problem at hand. Let me know, once you done some predictions or some place you are stuck $\endgroup$
    – Polymath
    Sep 2, 2022 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.