11
votes
What are the good parameter ranges for BERT hyperparameters while finetuning it on a very small dataset?
Data augumentation
If the number of text data is small, text data argumentations may be applicable e.g. nlpaug. Applying text summarization, removing stopwords or punctuations would be a simple way to ...
- 771
6
votes
Accepted
Which is the fastest image pretrained model?
The answer will depend on some things such as your hardware and the image you process. Additional, we should distinguish if you are talking about a single run through the network in training mode or ...
- 15k
5
votes
Accepted
Should you care about truncation and padding in an LLM even if it has a very large tokenizer.max_length so that truncation will never happen?
The large number you are seeing is not the maximum length, but the maximum representable integer at that precision. It's there because no maximum length has been set.
The original GPT-2 has a maximum ...
- 27.2k
3
votes
Understanding alpha parameter tuning in LORA paper
I had the same question. I still haven't got a convincing answer, but while searching I found these that might be helpful:
In this blog, they say:
Alpha scales the learned weights. Existing ...
3
votes
Accepted
How to download and fine tune a large language model locally?
Can I do it on a regular PC (I understand it may run for several hours)?
As a general guideline, you need a powerful workstation to perform this task. You should certainly consider having a cuda/gpu ...
- 56
3
votes
How to combine different models in Keras?
You can get the output of your models with model.output or get_layer
and combine them with ...
- 350
3
votes
Accepted
Since LoRA parameters are randomly initialized, shouldn't that mean that initially breaks a models output?
Not all the LoRA parameters are initialized randomly, only one of the matrices of the decomposition is. From the original LoRA article:
We use a random Gaussian initialization for A and zero for B, ...
- 27.2k
2
votes
Is it possible to add new vocabulary to BERT's tokenizer when fine-tuning?
To my understanding words unknown to the tokenizer will be masked with [UNKNOWN].
Your understanding is not correct.
BERT's vocabulary is defined not at word level, but at subword level. This means ...
- 27.2k
2
votes
Accepted
CNNs - Hyperparameter tuning with different training sizes of the same data set
First suggestion: you should first find a CNN architecture that satisfies you, and then stick with it.
Second suggestion: be careful with cross validation. CNNs are extremely "heavy" models, they can ...
- 6,315
2
votes
What are the good parameter ranges for BERT hyperparameters while finetuning it on a very small dataset?
How many classes do you have? Bert can handle a high-quality 12k dataset for binary classification. I recommend duplicating your positive test case 4x and sampling a 5k test cases from your negative ...
- 161
2
votes
Fine tuning Convolutional Neural Network with a learnable first layer
It's impossible to fine-tune ResNet with grayscale images directly because it was trained with color images from ImageNet.
There are 5 solutions:
Convert grayscale images to RGB images staying gray, ...
- 4,814
2
votes
How can you get a Huggingface fine-tuning model with the Trainer class from your own text where you can set the arguments for truncation and padding?
If you use transformers example scripts, for example, summarization, you can control padding and truncation with command line arguments --pad_to_max_length, ...
- 1,254
2
votes
Training a CNN in production on new data
This is a GOOD question. Say you have taken a pre-trained CNN model and then fine-tuned it on your data. The model is productionized and works well for a few weeks but now you observe a drift. The ...
- 908
1
vote
Accepted
Training model using BERT
You are overfitting A LOT.
This is usual when finetuning BERT on small datasets. I suggest you take a look at the BERT article to use it as a guidance for sensible hyperparameter values and finetuning ...
- 27.2k
1
vote
About improving the classifier when using a pre-trained model
Both of your models (with and with no extra layer) have high train accuracy but much lower scores on test data which indicates that both models overfit, i.e. they suffer from high variance (using the ...
- 5,450
1
vote
Accepted
LMM Fine Tuning - Supervised Fine Tuning Trainer (SFTTrainer) vs transformers Trainer
The short answer is that a Supervised Fine Tuning Trainer (SFTTrainer) is used for Instruct Fine Tuning. The HuggingFace library SFTTrainer has also support for training with QLoRA (4-bit Quantised ...
- 134
1
vote
Fine-tuning a pre-trained LLM for question-answering
Check the steps at the Huggingface beginner's guide Question Answering with SQuAD 2.0 to begin with a normal question answering model.
Have some look at the Fine-tuning guide at OpenAI as well (which ...
- 195
1
vote
How to limit a GPT chatbot in specific domain?
You can just include a clause in the prompt, before providing the user question, to make it kindly refuse to answer questions not related to the target domain.
- 27.2k
1
vote
How can I fine tune a model to detect digits, used to detect denominations of currency notes
This is a very common problem statement which can be solved using a Computer Vision model.
Is it possible to somehow fine-tune a digit-detection (in the wild) model (such as those trained using the ...
- 2,165
1
vote
Accepted
Pretrained vs. finetuned model
Even if both expressions are often considered the same in practice, it is crucial to draw a line between "reuse" and "fine-tune".
We reuse a model to keep some of its inner ...
- 4,814
1
vote
Is it okay to fine-tuning bert with large context for sequence classification?
There is a limiting factor here, which is the positional embeddings.
In BERT, positional embeddings are trainable (not sinusoidal) and support a maximum of 512 positions. To exceed such a sequence ...
- 27.2k
1
vote
Transformer similarity fine-tuned way too often predicts pairs as similar
For NLP related tasks the transformer tries it's best to match your output distribution but as with all ml tasks it will fail on some parts of your data. Your task is somewhat similar to bert's next ...
- 93
1
vote
Tuning a classifier for high precision, with no regard for recall
This is easy. Never call a case positive, and you’ll correctly classify all of the negative cases. Since you don’t care about missing positive cases, this should be fine.
If you have some tolerable, ...
- 4,044
1
vote
Accepted
Combining textual and numeric features into pre-trained Transformer BERT
When they are talking about aggregating their outputs, they mean the final embeddings (just before the classification layer) not the output of the network itself.
You take the embeddings from both of ...
- 712
1
vote
Fine tune the RetinaNet model in PyTorch
I am also trying to do a similar thing. The code below should work. After loading the pretrained weights on COCO dataset, we need to replace the classifier layer with our own.
...
- 133
1
vote
Accepted
Is it a good idea to combine fine tuning and feature extraction techniques?
The concept of multimodal learning is relevant: in this case, combining data from two modalities: 1) image signal using ResNet50 and 2) genomic features extracted from genes. Multimodal learning for ...
- 131
1
vote
Training Accuracy is getting higher, but Valid Loss and Accuracy is same every epoch
As you can see from the Train and Validation loss (and also accuracy). While your model is able to learn, your validation results do not improve. This means underfitting, or in this specific case, ...
- 1,168
1
vote
Does finetuning BERT involving updating all of the parameters or just the final classification layer?
By default, BERT fine-tuning involves learning a task-specific layer (For classification task, a neural network on top of the CLS token), as well as update the existing parameters of the model to ...
- 1,179
1
vote
Does finetuning BERT involving updating all of the parameters or just the final classification layer?
Both approaches are reasonable. Updating the BERT weights will train for longer period of time, but should give more accurate results.
- 934
Only top scored, non community-wiki answers of a minimum length are eligible