Skip to main content

Questions tagged [pretraining]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
20 views

How to choose ideal pretrained model for fine-tuning?

I started to work with LLMs lately and want to know how people choose their pre-trained models in their fine-tuning tasks? What is the criteria to choose the base model and which factors affect?
heyula's user avatar
  • 37
1 vote
1 answer
35 views

Is it methodologically correct to use the data to be used for finetuning in the pretrain phase of the BERT model?

Let us assume the training of a BERT model. An initial pre-train is performed with a large data set A. Subsequently a finetuning is performed with a dataset B which is part of A, but now with labels ...
Álvaro Loza's user avatar
1 vote
0 answers
36 views

Can unsupervised pretraining (autoencoders) be used for u-nets?

TLDR: Will a u-net pretrained as an autoencoder be able to learn a latent representation of the data if the encoder weights are frozen (can't game the system and pass forward the unmodified image)? ...
user avatar
1 vote
1 answer
63 views

How to choose (mean, std) for normalization in transfer learning?

I'm working on transfer learning based on ResNet50 pretrained model. Basically, I remove the last layer of ResNet50 and add new head layer. Then I train the model with my image dataset. Obviously, my ...
Morgan Cheng's user avatar
2 votes
1 answer
507 views

what is the difference between window size and context length of language model?

is window size and context length of language model one and the same thing? ******** following text is added as question with ONLY above text was not allowed ***** I am trying to understand how GPT ...
Vinay Sharma's user avatar
0 votes
0 answers
221 views

How to use pretrained encoder for customized Unet

if you have a standard Unet encoder such as resnet50, then it's easy to add pertaining to it. for example: ...
user836026's user avatar
7 votes
3 answers
4k views

Further Training a pre-trained LLM

My goal is to use the general knowledge and language understanding of a pre-trained LLM and to continue training on a smaller domain specific corpus to improve the model's knowledge on the domain. ...
Arthuro's user avatar
  • 101
0 votes
0 answers
73 views

This model is too slow. I'm looking for a good, fast-enough, out-of-the-box, pre-trained image classifier. Any tip?

I have been using this on a laptop without a GPU: https://github.com/pharmapsychotic/clip-interrogator Currently it takes about 10s to classify a single image on my own computer. I use ...
jokoon's user avatar
  • 101
1 vote
1 answer
205 views

Does the Transformer model has memory to store the state accross different data injection sequences(segments)?

I've trained a transformer model based on the pytorch tutorial: https://pytorch.org/tutorials/beginner/transformer_tutorial.html, But I found I've difficulties to understant this model's input and ...
Clock ZHONG's user avatar
1 vote
1 answer
314 views

Is there any concern for a pretrained model to overfitting to a fine-tuning task that has overlapping pretraining and training data?

Let's say my language model is pretrained on a general text corpus, and I want to use it for some specific downstream task that has it's datasets also included in the general corpus, is there any ...
Brian's user avatar
  • 11
0 votes
1 answer
22 views

How long is the generator pre-trained in SeqGAN?

I am reading up about SeqGAN and I am trying to understand the pretraining step better. The authors claim they want to maximize the Maximum Likelihood Estimation on the dataset S by pretraining the ...
postnubilaphoebus's user avatar
0 votes
1 answer
48 views

Reusing a model, pretrained on 19 classes, for just one of those classes

I have a pretrained net for semantic segmentation, which has been trained on the cityscapes dataset and its 19 classes (Person, car, traffic sign, …). One of those is "Person". I am only ...
J. Wu's user avatar
  • 1
1 vote
0 answers
7 views

Is there practice to train language-to-code transformer (multi-modal transformer) using uni-modal pretrained models-transformers?

Language-to-code transformation/generation require multiple skills - language and reasoning skills to digest the core problem from the natural language specification. And programming language ...
TomR's user avatar
  • 141
0 votes
2 answers
559 views

Pretrain RoBERTa model with new data using PyTorch library

I've pretrained the RoBERTa model with new data using a 'simpletransformers' library: ...
CapJS's user avatar
  • 125
1 vote
0 answers
268 views

Is it possible to "fine-tune" a pre-trained logistic regression model?

Fine tuning is a concept commonly used in deep learning. We may have a pre-trained model and then fine-tune it to our specific task. Does that apply to simple models, such as logistic regression? For ...
eduardokapp's user avatar
1 vote
1 answer
229 views

Pretrained vs. finetuned model

I have a doubt regarding terminology. When dealing with huggingface transformer models, I often read about "using pretrained models for classification" vs. "fine-tuning a pretrained ...
lazarea's user avatar
  • 289
2 votes
0 answers
94 views

how to improve recall by retraining a model on its feedback

I am creating a supervised model using sensitive and scarce data. For the sake of discussion, I've simiplified the problem statement by assuming that I'm creating a model for identifying dogs. Let's ...
learnlifelong's user avatar
0 votes
1 answer
92 views

Can I leave natural outliers in a dataset in training?

Can I leave unedited natural outliers in a dataset (outliers that have not appeared just because of mistyping of mistakes in the data)? Or should I also remove them or change them?
Zexxxx's user avatar
  • 33
1 vote
1 answer
300 views

test data is not a good representation of train data

I have predefined train and test sets. On generating some statistics like value_counts and checking the unique values, I feel that there is a 'lot' of difference between the distributions of the ...
letdatado's user avatar
  • 113
0 votes
0 answers
177 views

How long does it take to fine-tune XLNet?

XLNet takes a lot more time than BERT during pre-training. This results in XLNet performing better than BERT in over 20 NLP tasks. How long does XLNet take for fine-tuning (let's assume this is ...
Tony Jesuthasan's user avatar
0 votes
0 answers
28 views

Using mathematical derivatives of input data to augment training input data

I'm thinking of how to design a basic feedforward neural network that would be able to predict future datapoints given past datapoints. I'm very new to neural network design so I'm wondering if there'...
mhdnt's user avatar
  • 9
2 votes
1 answer
3k views

Fine-tuning pre-trained Word2Vec model with Gensim 4.0

With Gensim < 4.0, we can retrain a word2vec model using the following code: ...
NST's user avatar
  • 51
1 vote
0 answers
19 views

Working on an image classification project (microscopic images) , have some doubts [closed]

Currently, I am working on an image classification project. The data set contains very high resolution images taken via an electron microscope. Hence, I have few and limited instances. I have done EDA ...
Aditi's user avatar
  • 11
1 vote
1 answer
386 views

What the differences between self-supervised/semi-supervised in NLP?

GPT-1 mentions both Semi-supervised learning and Unsupervised pre-training but it seems like the same to me. Moreoever, "Semi-supervised Sequence Learning" of Dai and Le also more like self-...
Inhyeok Yoo's user avatar
0 votes
1 answer
43 views

What is the common practice for NLP or text mining for non-English?

A lot of natural language processing tools are pre-trained with corpus in English. What if ones need to analyze, say, Dutch text? The blogs I find online are mostly saying traslating text into English ...
Paw in Data's user avatar
0 votes
1 answer
879 views

Where to get models with weights instead of only weights? What's the purpose of .h5 files?

I have downloaded .h5 files from qubvel/resnet and qubvel/efficientnet. I was trying to use some models as a backbone for my model but I'm getting the following ...
karlosos's user avatar
  • 103
1 vote
2 answers
537 views

Would there be any reason to pretrain BERT on specific texts?

So the official BERT English model is trained on Wikipedia and BookCurpos (source). Now, for example, let's say I want to use BERT for Movies tag recommendation. Is there any reason for me to pretrain ...
Moradnejad's user avatar
1 vote
1 answer
809 views

How to access GPT-3, BERT or alike?

I am interested in accessing NLP models mentioned in scientific papers, to replicate some results and experiment. But I only see waiting lists https://openai.com/blog/openai-api/ and licenses granted ...
user305883's user avatar
0 votes
2 answers
126 views

Logic behind pre-trained weights and transfer learning

I am not sure about the logic behind how pre-trained weights make sense and translate into a new problem. To be more specific; for example, in a object detection network, how would a model's weights ...
Sahand's user avatar
  • 1
2 votes
2 answers
2k views

Does finetuning BERT involving updating all of the parameters or just the final classification layer?

Currently learning and reading about transformer models, I get that during the pretraining stage the BERT model is trained on a large corpus via MLM and NSP. But during finetuning, for example trying ...
spnc's user avatar
  • 21
1 vote
2 answers
4k views

Deploying multiple pre-trained model (tar.gz files) on Sagemaker in a single endpoint

We have followed the following steps: Trained 5 TensorFlow models in local machine using 5 different training sets. Saved those in .h5 format. Converted those into tar.gz (Model1.tar.gz,...Model5.tar....
Subh2608's user avatar
3 votes
1 answer
101 views

Are there any objections to using the same (unlabelled) data for pre-training of a BERT-Based model and the downstream task?

I'm looking to train an Electra model using unlabelled data in a specific field. Are there any objections to using the same data for unsupervised learning and then using the same data downstream for ...
user103134's user avatar
1 vote
2 answers
646 views

Semantic segmentation with greyscale images

I'm trying to reproduce a research with greyscale images instead of colour images. I have found that there are pre-trained networks, like VGG16, with ImageNet. But that dataset has colour images, and ...
VansFannel's user avatar
0 votes
1 answer
25 views

Question of pretraining text-generation task, it seems that pretraining is not work for a small model?

My task is to generate keywords from sentences. I pretrain a text-generation model. I mask the sentences' tokens and predict the whole sentences' tokens. Pretraining batch_size = 8 and step = 1000000 ...
惊天补扣's user avatar