11 votes

Is there any proven disadvantage of transfer learning for CNNs?

Based on my experience, not just for ImageNet, if you have enough data it's better to train your network from scratch. There are numerous reasons that I can explain why. First of all, I don't know ...
Green Falcon's user avatar
11 votes
Accepted

What are the consequences of not freezing layers in transfer learning?

I think that the main consequences are the following: Computation time: If you freeze all the layers but the last 5 ones, you only need to backpropagate the gradient and update the weights of the ...
David Masip's user avatar
  • 6,051
7 votes

What is the purpose of untrainable weights in Keras

One common application is to freeze an embedding layer. Freezing this layer will prevent the embedding from updating its weight which can be a desirable thing, especially for a text embedding layer. ...
Tophat's user avatar
  • 2,410
7 votes
Accepted

Default value of learning rate in adam optimizer - Keras

Learning rate is a very important hyperparameter, and often requires some experimentation. There are some good Related questions here, make sure to check those out. With too large a learning rate, ...
Ben Reiniger's user avatar
  • 11.7k
7 votes

Further Training a pre-trained LLM

Yes you are on the right track. What you are mentioning is called fine tuning the model. I personally have done this and used the same approach. The LLM I used was GPT-J 6B to generate MCQ's. Some ...
spectre's user avatar
  • 2,020
6 votes

What is the purpose of untrainable weights in Keras

We use freezing to employ transfer learning. Deep learning has a great hunger for data. In some tasks you may not have so much data, but there may already be a pre-trained network that can be helpful. ...
Green Falcon's user avatar
6 votes
Accepted

Which is the fastest image pretrained model?

The answer will depend on some things such as your hardware and the image you process. Additional, we should distinguish if you are talking about a single run through the network in training mode or ...
n1k31t4's user avatar
  • 14.8k
6 votes
Accepted

Effect of Stop-Word Removal on Transformers for Text Classification

Very interesting question. Easy, but probably lazy answer When using pre-trained models, it is always advised to feed it data similar to what it was trained with. Basically, if it matters, don't ...
Valentin Calomme's user avatar
5 votes
Accepted

Over fitting in Transfer Learning with small dataset

First of all: I think you should reduce the number of FC layers and number of nodes of FC layers, for example, one FC with 256 or 512, or 2 FC with 256 and 512. Try this. Try to make your batch size ...
Hunar's user avatar
  • 1,137
5 votes

How to use fine tuning of BERT when i have unlabelled dataset of text documents?

So, how should I refine the word/sentence embeddings vector given by the BERT model in the case when I have a set completely unlabelled set of documents? What are you looking to achieve with ...
Simon Larsson's user avatar
4 votes

What are the consequences of not freezing layers in transfer learning?

The reason it can save computation time is because your network would already be able to extract generic features from your dataset. The network will not have to learn extracting generic features from ...
vivek's user avatar
  • 41
4 votes
Accepted

Make the CNN to say "I don't know"

You could put a one-class classification model before your CNN. This would mean that you treat both your classes as one and then frame it as an anomaly detection problem. There are some different ...
Simon Larsson's user avatar
4 votes
Accepted

Are mainstream pre-trained models useful as discriminators?

Using a standard network architecture is perfectly reasonable. Most discriminator architectures are trivially different variants of well-known architectures anyway. Depending on the GAN loss, starting ...
user3658307's user avatar
  • 1,020
4 votes
Accepted

How many layers should I replace in transfer learning CNN

To build on the previous answer: In transfer learning, the goal is to use a pre-trained model and tweak the model to then specialise it to suit a certain task. So, what we do is, as SrJ has eluded to, ...
shepan6's user avatar
  • 1,428
4 votes

How to combine the features extracted from different CNN architectures?

You can tf.keras.layers.concatenate your extracted feature before flatten them in order to pass it to your fully connected layer. Note than you can actually use other operation such as multiply or add ...
Chopin's user avatar
  • 352
4 votes

Overfitting while fine-tuning pre-trained transformer

What makes you think your model is overfitting? Are you concerned about the difference between the training loss and validation loss? If so, this is not overfitting. Overfitting is when the weights ...
emily_learner's user avatar
4 votes

Overfitting while fine-tuning pre-trained transformer

Your question is valid. There are couple of known issues when trying to fit BERT-large version on small datasets (small implies a couple of 1000 training data points). The number of parameters itself ...
Allohvk's user avatar
  • 888
4 votes
Accepted

How to use BERT in seq2seq model?

In principle, it is possible to reuse the special tokens as you describe. However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain ...
noe's user avatar
  • 25.7k
3 votes

Over fitting in Transfer Learning with small dataset

I implemented various architectures for transfer learning and observed that models containing BatchNorm layers (e.g. Inception, ResNet, MobileNet) perform a lot worse (~30 % compared to >95 % test ...
mattseibl's user avatar
3 votes

Default value of learning rate in adam optimizer - Keras

There is a particular library called as ReduceLROnPlateau, that will reduce the learning rate, based on the factor value you mention. And this seems working good for all problem cases.
Monish Sakthi's user avatar
3 votes

How to add a new category to a existing trained deep learning model?

You cannot do that without re-training at least part of the model. You will have to replace the existing output layer with a new output layer that has the desired number of neurons. That, of course, ...
georg-un's user avatar
  • 1,231
3 votes

incremental learning vs transfer learning

Transfer Learning: for example you want to predict price of article normally we use previous data based on that we design model .while new data came still we use that model for prediction here we are ...
HEMANTHKUMAR GADI's user avatar
3 votes
Accepted

PyTorch: How to use pytorch pretrained for single channel image

I came across a code where the user had this very innovative method to tackle this problem. Here is the small trick to convert any pre-trained network to accept 1 channel images without loosing ...
thanatoz's user avatar
  • 2,405
3 votes

Imbalanced Dataset (Transformers): How to Decide on Class Weights?

The point of setting class weights is to manipulate the loss function to put more focus on the minor label. In fact, each of the data point passed to your learning algorithm will contribute ...
Quy Dinh's user avatar
3 votes
Accepted

Why are results without Transfer Learning better than with Transfer Learning?

As @fuwiak mentioned, transfer learning may not work if pre-trained model has been fitted on a "very different" dataset. Typically if the pre-trained network extract information that is not ...
etiennedm's user avatar
  • 1,385
3 votes

Is Flatten() layer in keras necessary?

With GlobalAveragePooling2D, only one Feature per Feature map is selected by averaging every elements of the Feature Map. e.g. if your global average pooling layer input is 220 x 220 x 30 you will ...
ashraful16's user avatar
3 votes
Accepted

Is Flatten() layer in keras necessary?

Although the first answer has explained the difference, I will add a few other points. If the model is very deep(i.e. a lot of Pooling) then the map size will become very small e.g. from 300x300 to ...
10xAI's user avatar
  • 5,574
3 votes

Can we fine-tune a model on the same dataset which it is pretrained on?

In the context of that paper, pre-train then fine-tune on the same dataset does not really make sense, as the pre-training is unsupervised, and the fine-tuning is with labelled data. But, generally, ...
Darren Cook's user avatar
2 votes

Why is input preprocessing in VGG16 in Keras not 1/255.0

The pre-trained weights that are available on Keras are trained with the preprocessing steps defined in preprocess_input() function that is made available for each ...
Arun Ponnusamy's user avatar
2 votes
Accepted

Transfer learning (on pre-trained inception net model) for multi label classification is giving similar probability for all labels

In my experience, the example code for a low number of classes (<200) works well. When moving to more classes the imbalance data makes the network converge to the same numbers. You have imbalance ...
jorgemf's user avatar
  • 211

Only top scored, non community-wiki answers of a minimum length are eligible