Suppose during testing it suggested with accuracy 40% for an image containing person of say class A. I would like to add this image back to the models training dataset and train the model adaptively, so that it improves the accuracy for this person. That would mean transferring an image from the test set to the training set.
Whilst adding data that appears important to your task to the training set will often improve performance, you have a major problem here. You must base your measure of performance on the test set. Altering the test set can invalidate your test results so far. And of course if you take a problem item out of the test set, your performance on the test set will go up even if you don't use the it to train with, because it was a result that was bringing down your average. So, you might improve your classifier, but at the expense of not knowing whether you have improved the classifier.
A good default position here is that the test set should not be altered. Once set aside, that data is for measuring performance of your tuned model, and should not be used for anything else, to avoid this measurement problem.
However, there are a couple of things you could do:
Error analysis
Look at the failing cases in the test set and try to figure out why they are being misclassified. For an image classification task, you might be able to figure out by viewing the image directly. Is there unusual lighting, pose, any different foreground or background objects? In the case of face recognition, has the subject got different hairstyle, makeup, hat, glasses or clothing?
If you spot something that's a one-off difference in your problem image, then that may be a sign that you need some more variety in your training data and should collect more. You should not collect it from the test data though, because then you will not be able to tell how well your classifier is doing at the task any more.
It is possible that investigating a failed item will show up something that means you want to alter or remove the test data item. This would happen for example if you decide it is not representative of how you want your model to be used. For instance, all the other images in the data set are from CCTV - which is how you intend it to be used - and the bad example is from a studio photo shoot. In that case, it may be worth the effort to clean up your test data and re-evaluate your best models so far.
K-fold cross validation
You can use k-fold cross validation to gain more accurate assessments of perfomance when tuning hyperparamaters (number of neurons, layer architecture). This doesn't technically replace a test set for measuring final performance, but it's not too bad a fallback position. An extreme version of this would be leave-onme-out cross validation.
In theory you could select your best hyperparameters using k-fold cross validation, have a good idea that this would be the best you could get, and a rough idea of what the generalisation performance was. Then train a final model using the same hyperparameters and all of your data.
You may be able to get away with this if you are happy to not know (or not need to report) an unbiased performance metric, but be reasonably certain you have close to the top performance out of the models that you have investigated.
Keep the test set as-is, do both the above, maybe collect more training data
I would recommend instead this as the best approach if you can afford the time and effort. For better or worse, you have set your goals for the first attempt at your project, when you selected the test set.
You can search harder for hyperparameters that generalise the best, using a small amount of data, by using k-fold cross validation. You can try to figure out why your accuracy is low by looking at the errors from the model and finding ways to fix them without using the test data in the training set.
Eventually you will have your best model (according to k-fold CV on the training data), and a measure of how it performs on the test set. The project is then done. Any further work to get even better accuracy would require some kind of re-thinking or start of a new project.
