I am trying to train a text classifier with open-source data to generalize on the real user traffic (henceforth "real data"). However, even though I have many annotated open-source data, I have only a few hundred real data.

I know the best way to generalize to the user traffic is by training a model with such data. However, given the scarcity of real data, it does not make sense to do so (I am aware that this may be possible with LLMs like LLaMA2, but I hope to work with BERT-like small models).

My idea is to train a model using open-source data (without training on the real data) that performs well on the real data, likely through some sort of regularization or constrained optimization on the real data.

However, I am not sure what options I have. Could anyone provide me with some pointers? Thank you!

  • 1
    $\begingroup$ If you have just "a few hundred real data" why waste time with ML models at all? Can you simply review all your data manually and record your observations in a Spreadsheet? If there is someone who needs to use your model with more data, you can negotiate with the them to provide you some data for training. $\endgroup$
    – Valentas
    Nov 10, 2023 at 13:34
  • $\begingroup$ @Valentas Thank you for your attention! You are right I could ask people to look at each sample. However, my intention is to learn something from the limited data and generalize to unseen data by applying the model to new data. $\endgroup$
    – Mr.Robot
    Nov 10, 2023 at 18:29
  • $\begingroup$ Can you elaborate on what "user traffic" means in this context? $\endgroup$
    – noe
    Nov 10, 2023 at 18:42
  • $\begingroup$ The data is basically legal texts we hire professionals to annotate. We could only afford to annotate a few hundred as annotating each is quite expensive. $\endgroup$
    – Mr.Robot
    Nov 12, 2023 at 6:05

4 Answers 4


Given that your finetuning dataset is very small, I suggest using multitask finetuning (multitask learning but in a finetuning situation):

  1. Download other legal datasets from the internet. If possible, choose datasets that support a task that is as compatible as possible with your task. For instance, if your task is classification, choose legal datasets meant for classification; if your task is labelling, choose legal datasets meant for labelling. You may try with only one dataset or try with many to see if you get further improvements.

  2. Combine them with your dataset and, for each task (your own task plus the newly added ones), add an extra marker to the text to let the model know which one it is (e.g. a prefix "TASK-1" for your task and "TASK-N" for the others); to represent this "task marker", you may try with a dedicated token or just plain text (i.e. subject to tokenization), which will be faster to test. To represent classes/labels, I suggest you add all possible outputs in a single table, without reusing indices.

  3. Finetune the model with the combined dataset.

  4. At inference time, you just always add the marker for your task (e.g. "TASK-1" in the previous example). To avoid generating classes/labels of the wrong task, you can simply mask the softmax.

With this approach, your model will benefit from the representations learned from other legal texts, which will hopefully improve your results in your original task.

This approach is common (not for finetuning but for training from scratch) in other scenarios like machine translation, for cases where you want to translate some language into a low resource language. You can combine your low-resource language data with other datasets of a closely related high-resource language (e.g. Portuguese and Spanish, Finnish and Estonian) to enlarge your training data, and you just let the model know which target language you want by adding a marker to the input.

  • $\begingroup$ Thank you for your detailed response. This idea is quite interesting. I have two questions: (1) do you have a relevant paper that used this idea before, and (2) does this scheme require training a new tokenizer? $\endgroup$
    – Mr.Robot
    Nov 13, 2023 at 21:16
  • $\begingroup$ (1) I have not seen this idea applied to finetuning; for machine translation, you can check this paper of mine. (2) It should not be necessary to train a new tokenizer. $\endgroup$
    – noe
    Nov 13, 2023 at 22:58
  • $\begingroup$ I will try to implement this idea and see how it works on my settings. $\endgroup$
    – Mr.Robot
    Nov 15, 2023 at 4:01

Given your question details, it seems you need to try transfer learning, where you reuse an existing pre-trained model on new data to fine tune the model generalization capabilities according to the new problem requirements (whatever problem you are trying to solve with your solution).

There are lots of options from which you can grab pre-trained models. For example, as of Nov 2023, Hugging face is a popular repository of pre-trained models.

  • $\begingroup$ Thanks for the response. But nowadays, using models from one of those available on HuggingFace is almost like a norm; this is also the case in my project: I almost always use RoBERTa-series models. I wonder what the options are when I already use these models and try to improve upon them. $\endgroup$
    – Mr.Robot
    Nov 12, 2023 at 5:59
  • $\begingroup$ Based on your new details, if you already use pre trained models like BERT (or its flavors) and HuggingFace, There are three options for you. (1) Trained a model from scratch (can take several days for training and many $$$ if trained on a public cloud like AWS). (2) create a new state-of-the-art ML model and its implementation. (3) use private unique data that has not been used before along with pretrained models. The intent of my original answer was to point out transfer learning as a method to address your question original requirements. $\endgroup$
    – Full Array
    Nov 14, 2023 at 12:54

You may consider the following options:

  1. Artificially enhance the real data through data augmentation. You can utilize libraries specifically designed for this purpose. This technique allows you to expand the real data significantly.

  2. Initially train your model using the open-source data and save your model. After that, fine-tune the saved model with the real data. During this fine-tuning phase, you can freeze all the layers except for the classification layer (the final layer) and conduct training solely on this layer. This technique is particularly effective when dealing with small datasets.


Fine-tune a pretrained model like BERT on your open-source data first. This will provide a strong baseline model. Then continue fine-tuning on the small real dataset. The pretrained weights will help prevent overfitting.

You can use BERT or RoBERTa. These models provide rich language representations that we can build upon:

import transformers

model = transformers.BertForSequenceClassification.from_pretrained("bert-base-uncased")

Next, we fine-tune this model on the open source dataset to adapt it to our text classification task:

# Load open source dataset
open_dataset = load_dataset(...)

# Fine-tune model 
trainer = transformers.Trainer(model=model)

This provides a strong baseline model. Now we need to make the most of our small real dataset. Here are some key techniques:

Data augmentation: Expand the size of the real dataset through augmentations like synonym replacement, random swap, deletion, etc. But don't alter semantics.

from nlpaug import Augmenter
augmenter = Augmenter(...) 

# Apply augmentations
augmented_dataset = augmenter.augment(real_dataset)

Transfer learning: Freeze pretrained weights, reinitialize classification layer, and fine-tune just that layer on the real data:

# Freeze pretrained weights
for param in model.base_model.parameters():
  param.requires_grad = False

# Reinitialize classifier  
model.classifier = nn.Linear(768, num_labels) 

# Fine-tune classifier only

Semi-supervised learning: Use unlabeled real data via techniques like pseudo-labeling:

# Generate pseudo-labels
pseudo_labels = model.predict(unlabeled_data)  

# Concatenate labeled and pseudo-labeled data
full_dataset = labeled_data + (unlabeled_data, pseudo_labels)

# Retrain model on combined dataset

Regularization: Use dropout, L1/L2 regularization, early stopping, etc. to prevent overfitting:

# Add dropout  
model.classifier = nn.Sequential(
  nn.Linear(768, num_labels)

# Use early stopping callback
trainer.train(real_data, callbacks=[EarlyStoppingCallback])

With the right combination of these techniques, we can maximize performance on the small real dataset while relying primarily on the open source pretraining. Thorough evaluation on a held-out real dataset set.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.