2
$\begingroup$

I'm still a bit new to deep learning. What I'm still struggling, is what is the best practice in re-training a good model over time?

I've trained a deep model for my binary classification problem (fire vs non-fire) in Keras. I have 4K fire images and 8K non-fire images (they are video frames). I train with 0.2/0.8 validation/training split. Now I test it on some videos, and I found some false positives. I add those to my negative (non-fire) set, load the best previous model, and retrain for 100 epochs. Among those 100 models, I take the one with lowest val_loss value. But when I test it on the same video, while those false positives are gone, new ones are introduced! This never ends, and Idk if I'm missing something or am doing something wrong.

How should I know which of the resulting models is the best? What is the best practice in training/re-training a good model? How should I evaluate my models?

Here is my simple model architecture if it helps:

def create_model():
  model = Sequential()
  model.add(Conv2D(32, kernel_size = (3, 3), activation='relu', input_shape=(300, 300, 3)))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Dropout(0.2))
  model.add(Flatten())
  model.add(Dense(256, activation='relu'))
  model.add(Dropout(0.2))
  model.add(Dense(64, activation='relu'))
  model.add(Dense(2, activation = 'softmax'))

  return model

#....
if retrain_from_prior_model == False:
    model = create_model()
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
else:
    model = load_model("checkpoints/model.h5")
$\endgroup$

3 Answers 3

1
$\begingroup$

In most cases, one shouldn't retrain a trained network with only the new data. Rather, train the network from scratch with the new and old data.

Adding new data and retraining the model just on that new set of data, will probably make your model fit to only that new data, thus forgetting general features from the other data it was trained on.

Also, instead of selecting your final model based on only validation loss, you should select your model based on a validation metric. For example, in your case, it could be accuracy, precision, recall etc

$\endgroup$
2
  • $\begingroup$ I don't retrain only on new data. The new data are added to the previous dataset so I retrain on the bigger dataset. $\endgroup$
    – Mary
    Feb 17, 2020 at 16:24
  • 1
    $\begingroup$ @Sid I would argue that you don't necessarily need a separate metric to evaluate your neural network. validation loss can be used as a metric. $\endgroup$
    – spectre
    May 14, 2023 at 5:18
1
$\begingroup$

The architecture and implementation of your model looks good, and you're doing well by using early stopping. However, you may want to consider using data augmentation to generate more training samples. Additionally, you can try fine-tuning a pre-trained model, such as ResNet, on your data to see if it improves your model's performance.

You can evaluate your models using metrics such as accuracy, precision, recall, F1-score, and ROC-AUC. These metrics can help you understand how well your model is performing and identify areas for improvement.

Here are some best practices:

Monitor your validation loss: As you're retraining your model, make sure to monitor your validation loss and accuracy. This will give you an idea of how well your model is generalizing to new data. If your validation loss starts increasing, it's an indication that your model is overfitting, and you should stop retraining the model.

Use early stopping: Early stopping is a technique that stops the training process when the validation loss stops improving. It can save you a lot of time and effort, especially if you have a large dataset. Keras provides an implementation of early stopping that you can use.

Use data augmentation: Data augmentation is a technique that generates new training samples by applying transformations to your existing data. This can help prevent overfitting and improve the generalization of your model.

Fine-tune your model: Fine-tuning is a technique where you take a pre-trained model and train it on your specific task. If your model is not performing well, you could try fine-tuning a pre-trained model on your data. This could help improve the performance of your model.

Use a learning rate scheduler: A learning rate scheduler is a technique that reduces the learning rate as the training progresses. This can help the model converge faster and prevent overfitting.

$\endgroup$
0
$\begingroup$

Your model is overfitting to the training data. You are adding more data to training data but the model is overfitting to that additional data.

To reduce overfitting, you need to increase regularization.

Common options:

  • Keep adding data to the training dataset until you cover all possible scenarios.
  • Add data augmentation.
  • Increase dropout.
$\endgroup$
1
  • $\begingroup$ Just curious, how did you come to the conclusion that the model is overfitting without the seeing metrics/results? Is it the fact that when the user trains on the FP cases, those are predicted correctly but on the test set the results are not good enough? $\endgroup$
    – spectre
    May 14, 2023 at 5:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.