Questions tagged [pretraining]
The pretraining tag has no usage guidance.
25
questions
0
votes
1
answer
19
views
Reusing a model, pretrained on 19 classes, for just one of those classes
I have a pretrained net for semantic segmentation, which has been trained on the cityscapes dataset and its 19 classes (Person, car, traffic sign, …). One of those is "Person". I am only ...
- 1
0
votes
0
answers
3
views
Is there practice to train language-to-code transformer (multi-modal transformer) using uni-modal pretrained models-transformers?
Language-to-code transformation/generation require multiple skills - language and reasoning skills to digest the core problem from the natural language specification. And programming language ...
- 131
0
votes
0
answers
15
views
Can I use pre-trained coco weights for medical lesion segmentation? Or should I train the network from scratch?
I am trying to understand in what cases I can benefit from pre-trained weights. Sometimes, pre-trained weights works (can be fine-tuned) for other domains but I cannot understand how to make a ...
- 1
0
votes
0
answers
109
views
Pretrain RoBERTa model with new data using PyTorch library
I've pretrained the RoBERTa model with new data using a 'simpletransformers' library:
...
- 115
1
vote
0
answers
40
views
Is it possible to "fine-tune" a pre-trained logistic regression model?
Fine tuning is a concept commonly used in deep learning. We may have a pre-trained model and then fine-tune it to our specific task.
Does that apply to simple models, such as logistic regression?
For ...
- 111
1
vote
1
answer
55
views
Pretrained vs. finetuned model
I have a doubt regarding terminology. When dealing with huggingface transformer models, I often read about "using pretrained models for classification" vs. "fine-tuning a pretrained ...
- 289
2
votes
0
answers
53
views
how to improve recall by retraining a model on its feedback
I am creating a supervised model using sensitive and scarce data. For the sake of discussion, I've simiplified the problem statement by assuming that I'm creating a model for identifying dogs.
Let's ...
0
votes
0
answers
21
views
Baseline model and transfer learning
I've tried to find any guidance on using transfer learning when building baseline models for ML projects (CNN in my case) but found no clues on good practices in the matter.
My logic says that no ...
0
votes
1
answer
31
views
Can I leave natural outliers in a dataset in training?
Can I leave unedited natural outliers in a dataset (outliers that have not appeared just because of mistyping of mistakes in the data)? Or should I also remove them or change them?
- 33
1
vote
1
answer
93
views
test data is not a good representation of train data
I have predefined train and test sets. On generating some statistics like value_counts and checking the unique values, I feel that there is a 'lot' of difference between the distributions of the ...
- 13
0
votes
0
answers
61
views
How long does it take to fine-tune XLNet?
XLNet takes a lot more time than BERT during pre-training. This results in XLNet performing better than BERT in over 20 NLP tasks. How long does XLNet take for fine-tuning (let's assume this is ...
0
votes
0
answers
16
views
Using mathematical derivatives of input data to augment training input data
I'm thinking of how to design a basic feedforward neural network that would be able to predict future datapoints given past datapoints. I'm very new to neural network design so I'm wondering if there'...
- 9
2
votes
1
answer
948
views
Fine-tuning pre-trained Word2Vec model with Gensim 4.0
With Gensim < 4.0, we can retrain a word2vec model using the following code:
...
- 51
1
vote
0
answers
16
views
Working on an image classification project (microscopic images) , have some doubts [closed]
Currently, I am working on an image classification project. The data set contains very high resolution images taken via an electron microscope. Hence, I have few and limited instances.
I have done EDA ...
- 11
1
vote
1
answer
85
views
What the differences between self-supervised/semi-supervised in NLP?
GPT-1 mentions both Semi-supervised learning and Unsupervised pre-training but it seems like the same to me. Moreoever, "Semi-supervised Sequence Learning" of Dai and Le also more like self-...
- 33
0
votes
1
answer
25
views
What is the common practice for NLP or text mining for non-English?
A lot of natural language processing tools are pre-trained with corpus in English. What if ones need to analyze, say, Dutch text? The blogs I find online are mostly saying traslating text into English ...
- 145
0
votes
1
answer
265
views
Where to get models with weights instead of only weights? What's the purpose of .h5 files?
I have downloaded .h5 files from qubvel/resnet and qubvel/efficientnet. I was trying to use some models as a backbone for my model but I'm getting the following ...
- 103
1
vote
2
answers
345
views
Would there be any reason to pretrain BERT on specific texts?
So the official BERT English model is trained on Wikipedia and BookCurpos (source).
Now, for example, let's say I want to use BERT for Movies tag recommendation. Is there any reason for me to pretrain ...
- 235
0
votes
1
answer
303
views
How to access GPT-3, BERT or alike?
I am interested in accessing NLP models mentioned in scientific papers, to replicate some results and experiment.
But I only see waiting lists https://openai.com/blog/openai-api/ and licenses granted ...
- 155
0
votes
2
answers
60
views
Logic behind pre-trained weights and transfer learning
I am not sure about the logic behind how pre-trained weights make sense and translate into a new problem.
To be more specific; for example, in a object detection network, how would a model's weights ...
- 1
1
vote
2
answers
827
views
Does finetuning BERT involving updating all of the parameters or just the final classification layer?
Currently learning and reading about transformer models, I get that during the pretraining stage the BERT model is trained on a large corpus via MLM and NSP. But during finetuning, for example trying ...
- 11
1
vote
2
answers
3k
views
Deploying multiple pre-trained model (tar.gz files) on Sagemaker in a single endpoint
We have followed the following steps:
Trained 5 TensorFlow models in local machine using 5 different training sets.
Saved those in .h5 format.
Converted those into tar.gz (Model1.tar.gz,...Model5.tar....
- 13
3
votes
1
answer
69
views
Are there any objections to using the same (unlabelled) data for pre-training of a BERT-Based model and the downstream task?
I'm looking to train an Electra model using unlabelled data in a specific field. Are there any objections to using the same data for unsupervised learning and then using the same data downstream for ...
- 31
1
vote
2
answers
341
views
Semantic segmentation with greyscale images
I'm trying to reproduce a research with greyscale images instead of colour images.
I have found that there are pre-trained networks, like VGG16, with ImageNet. But that dataset has colour images, and ...
- 229
0
votes
1
answer
17
views
Question of pretraining text-generation task, it seems that pretraining is not work for a small model?
My task is to generate keywords from sentences.
I pretrain a text-generation model. I mask the sentences' tokens and predict the whole sentences' tokens.
Pretraining batch_size = 8 and step = 1000000
...
- 591