0
$\begingroup$

Problem Statement: To be able to detect faces of specific people with good accuracy and tag those images with the names of the people it contains.

To do this we have about 40 images person, and 10 classes.

I have been using Facenet model ( https://github.com/davidsandberg/facenet ) and I am getting good results after training the model with 10 classes & 40 images per class. I have used 100 images for testing.

The current hurdle im facing is : Suppose during testing it suggested with accuracy 40% for an image containing person of say class A. I would like to add this image back to the models training dataset and train the model adaptively, so that it improves the accuracy for this person. That would mean transferring an image from the test set to the training set.

Is it possible to have an adaptive learning added to my current solution? Not sure if I made my problem clear , I could explain more in detail if needed.

$\endgroup$
2
  • $\begingroup$ Hi, could you give more details about how you know the image contains A? Is it ground truth? Is this a new image added to the system after a first round of training (perhaps during a test), or is it one of the existing 40 images? Do you have separate test data (images that you do not train on ever, but use just to evaluate how well the model generalises instead of just memorising the training data), if so, how much? Could you clarify whether your idea is to attempt to use the test data to improve training (move an image from test data to training data), where tests show incorrect results? $\endgroup$ Sep 10, 2019 at 13:35
  • $\begingroup$ The image contains person A (ground truth, this image was not fed into the system for training, as in the model has not yet seen this image.) The image is the one for testing, this image is a part of the test dataset. among the given 100 images for testing, this image is having low accuracy percentage. And yes i want to know how to add this data back to the model and train it so that it improves accuracy. $\endgroup$
    – Arun Dev
    Sep 10, 2019 at 14:07

1 Answer 1

1
$\begingroup$

Suppose during testing it suggested with accuracy 40% for an image containing person of say class A. I would like to add this image back to the models training dataset and train the model adaptively, so that it improves the accuracy for this person. That would mean transferring an image from the test set to the training set.

Whilst adding data that appears important to your task to the training set will often improve performance, you have a major problem here. You must base your measure of performance on the test set. Altering the test set can invalidate your test results so far. And of course if you take a problem item out of the test set, your performance on the test set will go up even if you don't use the it to train with, because it was a result that was bringing down your average. So, you might improve your classifier, but at the expense of not knowing whether you have improved the classifier.

A good default position here is that the test set should not be altered. Once set aside, that data is for measuring performance of your tuned model, and should not be used for anything else, to avoid this measurement problem.

However, there are a couple of things you could do:

Error analysis

Look at the failing cases in the test set and try to figure out why they are being misclassified. For an image classification task, you might be able to figure out by viewing the image directly. Is there unusual lighting, pose, any different foreground or background objects? In the case of face recognition, has the subject got different hairstyle, makeup, hat, glasses or clothing?

If you spot something that's a one-off difference in your problem image, then that may be a sign that you need some more variety in your training data and should collect more. You should not collect it from the test data though, because then you will not be able to tell how well your classifier is doing at the task any more.

It is possible that investigating a failed item will show up something that means you want to alter or remove the test data item. This would happen for example if you decide it is not representative of how you want your model to be used. For instance, all the other images in the data set are from CCTV - which is how you intend it to be used - and the bad example is from a studio photo shoot. In that case, it may be worth the effort to clean up your test data and re-evaluate your best models so far.

K-fold cross validation

You can use k-fold cross validation to gain more accurate assessments of perfomance when tuning hyperparamaters (number of neurons, layer architecture). This doesn't technically replace a test set for measuring final performance, but it's not too bad a fallback position. An extreme version of this would be leave-onme-out cross validation.

In theory you could select your best hyperparameters using k-fold cross validation, have a good idea that this would be the best you could get, and a rough idea of what the generalisation performance was. Then train a final model using the same hyperparameters and all of your data.

You may be able to get away with this if you are happy to not know (or not need to report) an unbiased performance metric, but be reasonably certain you have close to the top performance out of the models that you have investigated.

Keep the test set as-is, do both the above, maybe collect more training data

I would recommend instead this as the best approach if you can afford the time and effort. For better or worse, you have set your goals for the first attempt at your project, when you selected the test set.

You can search harder for hyperparameters that generalise the best, using a small amount of data, by using k-fold cross validation. You can try to figure out why your accuracy is low by looking at the errors from the model and finding ways to fix them without using the test data in the training set.

Eventually you will have your best model (according to k-fold CV on the training data), and a measure of how it performs on the test set. The project is then done. Any further work to get even better accuracy would require some kind of re-thinking or start of a new project.

$\endgroup$
4
  • $\begingroup$ Hey, Thanks for the answer clears up a lot of questions I had. One thing that is still not making sense is : Now i have a class A trained with 100 images and tested upon another 100 images and i am satisfied with the results. Now i use this model in real time scenario and while recognizing the faces it shows the result as 30% probability of class A (which is correct by ground truth) now i wanted to know is there a way to feed in such data which are having low accuracy back to training such that the model will improve from then on.? $\endgroup$
    – Arun Dev
    Sep 11, 2019 at 3:33
  • $\begingroup$ @ArunDev: That would be a different scenario to your question. You have collected new data after deploying your model, and want to make improvements. There are two basic approaches: (1) verify and collect together your new data, then re-do the project - it will be quicker as you already have a good idea of hyperparameters. (2) continue training online. Neural networks support online training, but it is risky because they will likely forget older training data, gaining performance on recent data but losing it on older data. $\endgroup$ Sep 11, 2019 at 6:23
  • $\begingroup$ I don't know much about online training for image classifiers, perhaps there is a solution for you that reduces risk of the NN forgetting older data. If it interests you, perhaps ask a new question about that. $\endgroup$ Sep 11, 2019 at 6:26
  • $\begingroup$ Thanks, will look into online training for image classifiers and if my model supports it. $\endgroup$
    – Arun Dev
    Sep 11, 2019 at 6:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.