I would like to use a YOLOv3 network,load in the preexisting trained weights, then retrain the ending layer to recognize say, 20 labels.

Would adding labels to a single network in training reduce its accurary?

Would it be better to train multiple models:


Model A : recognizes 5 labels (ferrari,honda,ford,toyota,tesla)
Model B : regonzies 10 labels (store front / stopsign, speed sign, super charger station , etccc. )
Model C : recongizes 5 labels (pedestrian, dog, marathon runner, bicycler, scooter rider)

OR a single model

Model D: Recognizes 20 labels (all labels above put together)

Would Model D be as accurate as Model A,B,C put together?


1 Answer 1


It depends specially on data size and model capacity.

Therefore, we cannot give a definite "yes" or "no" prior to any experiment. However, we can enumerate possible cases, causes, and ways to improve, regardless of any specific model or dataset.

  1. If the number of data points per category is insufficient, which is frequently the case in reality, a single model for combined categories performs better, since the effect of data shortage will be attenuated by sharing a model among different but related tasks, i.e. shared knowledge. However, we should not forget about the capacity of model, if model capacity is not enough for the combined task (which depends on the architecture and size of the model), this sharing would backfire as the model spreads its limited capacity across multiple tasks and becomes "jack of all trades, and master of none". Therefore, this approach is favorable if model capacity permits, which is less of a concern for neural networks with flexible size constraints.

  2. If the number of data points per category is sufficient, which is rarely the case, or model capacity is limited (for example, when model performance stops improving when trained on more than 10% of data), single model per category is better since no capacity will be wasted on different (although related) tasks. From a different point of view, although this choice might be less favorable in terms of model accuracy, but it allows parallel development on smaller tasks, i.e. faster production, which could be a justified benefit for the cost we pay in terms of model accuracy.

Both approaches

We can also combine the two approaches. First we train a model on combined categories, second we feed a part (or parts) of the trained model (such as a layer, the final prediction, etc.) to a specific model that will be trained on one category. This is exactly what a pre-trained model does for us, it is trained on a more general task with more data (possibly even a different but related task), and then we use it for a specific task.

In your case, you can go for three levels:

  1. The pre-trained YOLO which is already built,
  2. Training MyYOLO on all categories (for example, by using one input from the last layer of YOLO, and one input directly from image), and
  3. Training MyYOLO2 on specific categories (for example, by using one input from MyYOLO and one input directly from image). Noting that at each level, weights of previous model remain constant to serve a constant input.
  • $\begingroup$ I am planning on using the pretrained yolov3 weights $\endgroup$
    – Ryu S.
    Apr 10, 2019 at 0:14
  • $\begingroup$ @RyuS. Nice. We cannot give a definite "yes" or "no", I updated the answer in this regard. Hope it helps. $\endgroup$
    – Esmailian
    Apr 10, 2019 at 12:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.