8
votes
Further Training a pre-trained LLM
Yes you are on the right track. What you are mentioning is called fine tuning the model. I personally have done this and used the same approach.
The LLM I used was GPT-J 6B to generate MCQ's. Some ...
- 2,165
4
votes
Accepted
what is the difference between window size and context length of language model?
Yes, in large language models, window and context length refer to the same thing: the maximum token sequence length that the language model can handle at once.
- 27.2k
3
votes
Accepted
How to access GPT-3, BERT or alike?
OpenAI has not released the weights of GPT-3, so you have to access it through their API. However, all other popular models have been released and are easily accessible. This includes GPT-2, BERT, ...
- 27.2k
2
votes
Accepted
Where to get models with weights instead of only weights? What's the purpose of .h5 files?
There are several options when saving and loading a keras model, as explained at https://www.tensorflow.org/guide/keras/save_and_serialize:
save the whole configuration, including the architecture, ...
- 2,724
2
votes
Would there be any reason to pretrain BERT on specific texts?
BERT is a fairly large model that requires many data and lots of training time to achieve its state-of-the-art performance. More often than not, there isn't enough data nor resources to completely ...
- 181
2
votes
Accepted
Would there be any reason to pretrain BERT on specific texts?
Sure, if you have a large and good quality in-domain dataset, the results may certainly be better than with the generic pretrained BERT.
This has already been done before: BioBERT is a BERT model ...
- 27.2k
2
votes
Accepted
Deploying multiple pre-trained model (tar.gz files) on Sagemaker in a single endpoint
I also have been looking for answers regarding this before, and after several days of trying with my friend, we manage to do it. I attach some code snippet that we use, you may modify it according to ...
- 36
2
votes
Accepted
test data is not a good representation of train data
Here's is my attempt to answer these questions:
Avoid taking any insights from test_data. Do changes WRT the insights taken from the train_set only. However, every change I make should be replicated ...
- 387
2
votes
Accepted
Does the Transformer model has memory to store the state accross different data injection sequences(segments)?
For context, we are talking here about a language modeling task, that is, predicting the next word or token.
In normal transformers, there is no “state transfer” between different segments (either ...
- 27.2k
1
vote
How to choose ideal pretrained model for fine-tuning?
There are too many! Some examples:
Intended use of the model regarding its compatibility with the licenses of available model (p.ej. is commercial use allowed?).
intended use of the model to know if ...
- 27.2k
1
vote
Accepted
Is it methodologically correct to use the data to be used for finetuning in the pretrain phase of the BERT model?
I'd say that it's correct.
BERT pre-training doesn't use labels, as it uses two self-supervised objectives:
masked language model (mask a word in the middle of a sentence, and guess what it is)
next ...
- 6,096
1
vote
How to choose (mean, std) for normalization in transfer learning?
The normalization step is meant to provide input values to your network that have mean 0 and variance 1. For this, and given that you are going to provide your own data at fine-tuning and inference ...
- 27.2k
1
vote
Accepted
Pretrained vs. finetuned model
Even if both expressions are often considered the same in practice, it is crucial to draw a line between "reuse" and "fine-tune".
We reuse a model to keep some of its inner ...
- 4,814
1
vote
Fine-tuning pre-trained Word2Vec model with Gensim 4.0
You can try the following steps to fine-tune on your domain-specific corpus using Gensim 4.0:
Create a Word2Vec model with the same vector size as the pretrained model
...
- 111
1
vote
Accepted
What the differences between self-supervised/semi-supervised in NLP?
Semi-supervised learning is having label for a fraction of data, but in self-supervised there is no label available. Imagine a huge question/answer dataset. No one labels that data but you can learn ...
- 6,622
1
vote
What is the common practice for NLP or text mining for non-English?
Translation as a pre-processing step is usually sufficient for many tasks (e.g. sentiment classification), but naturally undesirable for other tasks e.g. grading someone in written Dutch fluency.
...
- 66
1
vote
Are there any objections to using the same (unlabelled) data for pre-training of a BERT-Based model and the downstream task?
Not at all. A recent ACL paper by AllenAI even says this is the best way. They recommend continuing pre-training on the task data and claim that it reduces the problems caused by domain mismatch. So, ...
- 1,771
1
vote
Logic behind pre-trained weights and transfer learning
In the case of CNN, you are correct that you cannot use the final layer weights if the number of categories is different. But you CAN reuse the weights in the initial layers. These recognise the lower-...
- 908
1
vote
Logic behind pre-trained weights and transfer learning
If we classify new objects using transfer learning:
We delete the top Dense layer of the pre-trained neural network.
Now suppose you have to classify 5 classes, so your final dense layer will contain ...
- 707
1
vote
Does finetuning BERT involving updating all of the parameters or just the final classification layer?
By default, BERT fine-tuning involves learning a task-specific layer (For classification task, a neural network on top of the CLS token), as well as update the existing parameters of the model to ...
- 1,179
1
vote
Does finetuning BERT involving updating all of the parameters or just the final classification layer?
Both approaches are reasonable. Updating the BERT weights will train for longer period of time, but should give more accurate results.
- 934
Only top scored, non community-wiki answers of a minimum length are eligible