1
$\begingroup$

I have 2 instances of an object detection model. The only difference between these two models is the training data used:

  1. The first model was trained with a small training set
  2. The second model was trained on a larger training set than the first one

The first model was trained on the following hyperparameters:

  • Number of iterations: 250k
  • Batch Size: 10
  • Learning Rate: warms up to 0.001 and decreases to 0.0002 after 150k iterations

Since the second model has more training data, I assumed I need to change the hyperparameters a bit. So I tried training the second model on the following hyperparamters:

  • Number of iterations: 600k
  • Batch Size: 10
  • Learning Rate: warms up to 0.001 and decreases to 0.0002 after 400k iterations

When I measure the mAP for both models on a testing set, the first model vastly outperforms the second model.

model 1 mAP: 0.924

model 2 mAP: 0.776

This leads me to my question for you:

How would the hyperparameters (batch size, learning rate etc) change when the size of your training set increases? What factors need to be considered for this increase in training set size, in order to get the most optimal model possible?

Any and all responses will be greatly helpful. Thank you :)

$\endgroup$
0
$\begingroup$

A major difference between the first and the second model you trained is the size of the data assuming that the model is not pretrained. Increased data, of course, needs increased epochs. According, the batch size must also increase.

Batch Size:

  • While training on the smaller dataset, a batch size of 10 yielded better results. The errors were averaged over 10 samples and then back-propagated through the model. But now for the larger dataset, the batch size remains the same and hence only little optimization occurs as the error is averaged over 10 samples only ( which is relatively smaller for a large dataset ).

Learning Rate:

  • For the larger dataset, the number of epochs is increased. The purpose of the learning rate is to scale the gradients of the loss with respect to the parameter. A smaller learning rate always helps as it prevents the overshooting of the minima of the loss function. I would insist you increase the learning rate so that the optimization does not diminish as we are having a larger number of epochs. Gradually decrease the learning rate, as the loss approaches its minima.

If you are training a popular architecture ( like Inception, VGG, etc. ) and that too on datasets like ImageNet, COCO with little modifications, you should definitely read the research papers published on various problems as they would provide a better start to the training.

| improve this answer | |
$\endgroup$
  • $\begingroup$ So even though I increased the number of epochs, the updates made weren't big enough because of the small batch size and learning rate. This makes a lot of sense. Since an increased batch size results in a more accurate gradient, can I assume that the learning rate can be increased without detriment? Or maybe consider increasing the batch size but use the same learning rate $\endgroup$ – Badhreesh M Rao Feb 5 at 16:19
  • $\begingroup$ Increase the learning rate so that the learning does not diminish. Then, decrease it gradually as the training progresses. $\endgroup$ – Shubham Panchal Feb 5 at 23:51
  • $\begingroup$ It's not clear to me that more data should necessitate more epochs; each data point will get run through the network the same number of times. And too small a learning rate risks getting stuck in a local minimum. Might more data also necessitate/suggest a different (larger) network architecture? $\endgroup$ – Ben Reiniger Feb 6 at 3:15
  • $\begingroup$ @BenReiniger If the batch size is the same in both cases, for the same number of training iterations, that will translate to different number of epochs. For example, say the first and second dataset has 5k and 10k images respectively. Batch size is 10 in both cases and you run your model for 250k iterations. First model runs for (10 * 250k)/5k = 500 epochs while second model runs for (10 * 250k)/10k = 250 epochs. This means each data point is NOT run through the model the same number of times. To correct this, you need to increase the batch size, meaning the model will run for more epochs. $\endgroup$ – Badhreesh M Rao Feb 6 at 9:24
  • $\begingroup$ With regards to architecture, you dont know the networks full capacity to learn the more complex data distribution from the second dataset yet. Only way to find out is to test it. If after a lot of hyperparameter tuning it still doesn't improve, then maybe consider using a bigger model $\endgroup$ – Badhreesh M Rao Feb 6 at 9:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.