Questions tagged [pretraining]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

How long does it take to fine-tune XLNet?

XLNet takes a lot more time than BERT during pre-training. This results in XLNet performing better than BERT in over 20 NLP tasks. How long does XLNet take for fine-tuning (let's assume this is ...
0
votes
0answers
11 views

Using mathematical derivatives of input data to augment training input data

I'm thinking of how to design a basic feedforward neural network that would be able to predict future datapoints given past datapoints. I'm very new to neural network design so I'm wondering if there'...
0
votes
0answers
9 views

One single-batch training on Huggingface Bert model "ruins" the model

For some reason, I need to do further (2nd-stage) pre-training on Huggingface Bert model, and I find my training outcome is very bad. After debugging for hours, surprisingly, I find even training one ...
2
votes
0answers
164 views

Fine-tuning pre-trained Word2Vec model with Gensim 4.0

With Gensim < 4.0, we can retrain a word2vec model using the following code: ...
1
vote
0answers
14 views

Working on an image classification project (microscopic images) , have some doubts [closed]

Currently, I am working on an image classification project. The data set contains very high resolution images taken via an electron microscope. Hence, I have few and limited instances. I have done EDA ...
0
votes
0answers
33 views

Does NSP task corrupt context during pre-training?

During the pre-training of BERT, if we just use MLM our input will be: [CLS] SentenceA [SEP]. So if there is a masked token in Sentence A, it will be predicted by ...
1
vote
1answer
31 views

What the differences between self-supervised/semi-supervised in NLP?

GPT-1 mentions both Semi-supervised learning and Unsupervised pre-training but it seems like the same to me. Moreoever, "Semi-supervised Sequence Learning" of Dai and Le also more like self-...
0
votes
1answer
18 views

What is the common practice for NLP or text mining for non-English?

A lot of natural language processing tools are pre-trained with corpus in English. What if ones need to analyze, say, Dutch text? The blogs I find online are mostly saying traslating text into English ...
0
votes
1answer
43 views

Where to get models with weights instead of only weights? What's the purpose of .h5 files?

I have downloaded .h5 files from qubvel/resnet and qubvel/efficientnet. I was trying to use some models as a backbone for my model but I'm getting the following ...
1
vote
2answers
174 views

Would there be any reason to pretrain BERT on specific texts?

So the official BERT English model is trained on Wikipedia and BookCurpos (source). Now, for example, let's say I want to use BERT for Movies tag recommendation. Is there any reason for me to pretrain ...
0
votes
0answers
12 views

(Pre)training an Embedding Layer via prediction (as an alternative to similarity) - does that makes sense?

Similarity is the go-to way to train embeddings - use a similarity matrix (eg dot product) between the embeddings of two inputs, train to increase it for connected inputs and decrease it for inputs ...
0
votes
1answer
173 views

How to access GPT-3, BERT or alike?

I am interested in accessing NLP models mentioned in scientific papers, to replicate some results and experiment. But I only see waiting lists https://openai.com/blog/openai-api/ and licenses granted ...
0
votes
0answers
30 views

Pretrained CNN model on animal dataset ( turtles images if exist )

I was wondering if there is a pretrained CNN model on an animal dataset. I am participating in a turtle face detection competition and was wondering if there was a pretrained image model to fine tune ...
0
votes
2answers
40 views

Logic behind pre-trained weights and transfer learning

I am not sure about the logic behind, how pre-trained weights actually make sense and translate into a new problem. To be more specific; for example in a object detection network, how would a model's ...
1
vote
2answers
417 views

Does finetuning BERT involving updating all of the parameters or just the final classification layer?

Currently learning and reading about transformer models, I get that during the pretraining stage the BERT model is trained on a large corpus via MLM and NSP. But during finetuning, for example trying ...
1
vote
1answer
1k views

Deploying multiple pre-trained model (tar.gz files) on Sagemaker in a single endpoint

We have followed the following steps: Trained 5 TensorFlow models in local machine using 5 different training sets. Saved those in .h5 format. Converted those into tar.gz (Model1.tar.gz,...Model5.tar....
3
votes
1answer
41 views

Are there any objections to using the same (unlabelled) data for pre-training of a BERT-Based model and the downstream task?

I'm looking to train an Electra model using unlabelled data in a specific field. Are there any objections to using the same data for unsupervised learning and then using the same data downstream for ...
1
vote
2answers
144 views

Semantic segmentation with greyscale images

I'm trying to reproduce a research with greyscale images instead of colour images. I have found that there are pre-trained networks, like VGG16, with ImageNet. But that dataset has colour images, and ...
0
votes
1answer
17 views

Question of pretraining text-generation task, it seems that pretraining is not work for a small model?

My task is to generate keywords from sentences. I pretrain a text-generation model. I mask the sentences' tokens and predict the whole sentences' tokens. Pretraining batch_size = 8 and step = 1000000 ...