I trained a convolutional neural network with a large (>90k) dataset of images. Since there is some confusion between two classes, I collected all the images that were misclassified and I augmented them with several methods. Furthermore, I collected new images that were not present in the original dataset. Now I would like to re-train my network in order to reduce the confusion. What is the best approach? Should I perform fine-tuning using only the new images (case A) or should I add the new images to the whole training set and re-train the network from zero (case B)?
My concerns are the following: in case A, most of the images belong to the same class, therefore the dataset would be very unbalanced. In case B, I am afraid that the impact of the new images on the whole dataset would be very small.