I would like to detect defects/anomalies in images. Due to the lack of images with anomalies, I try to solve the problem in an unsupervised manner. Until now, I trained a variational autoencoder together with an generative adversarial network with “good” images. Now, I have an encoder/decoder network which is able to produce images without anomalies. Here are two examples:

original image | cropped image & input of the encoder | output of the decoder

enter image description here

enter image description here

In those images, if the object is rotated (not vertical), then it is an anomaly (like the second image). To measure the difference between the input/output of the encoder/decoder network, I tried the structural similarity metric SSIM. However, the result is not satisfying enough as many images without an anomaly also have a low similarity value. Also the pixel-wise MAE or MSE metrics are not good as they do not look specifically on the structure and are very sensitive to the brightness. Here is the result with the SSIM metric (green are the good examples, red are the anomalies):

enter image description here

What I want to avoid is to add specific rules for the detection, in this example to detect vertical lines because there are many other objects with different anomalies which I would like to detect.

Do you have a suggestion for another image similarity metric? Am I on the right path or are there other/better methods to do unsupervised anomaly detection?

PS: There are only a few images with anomalies (<10 images), but many without an anomaly..

  • $\begingroup$ Do you use pooling? $\endgroup$
    – Lau
    Nov 24, 2018 at 11:02
  • $\begingroup$ I would love to have the dataset you are working on. Thanks :) $\endgroup$
    – Ali Fessi
    Mar 9, 2019 at 14:20
  • $\begingroup$ You can have a look into the literature, e.g. "Outlier Analysis" $\endgroup$ Oct 9, 2020 at 23:57

2 Answers 2


There seems to be a clear line between samples with anomalies and those without. Label them and run a classifier? e.g SVM. One shouldn't use more complex models than needed.

Also, be a bit careful with trying to detect all kind of anomalies. It's not always realistic and might lead you away from more straightforward, and perhaps better solutions.

You could also try something as simple as a histogram from one part of the image. Looks like it should be able to find the anomalies.


One thing you could try, would be to add a second loss term that uses also the latent representation of the VAE. The idea being that anomalies should also be outliers in latent space. That way you would have another channel that provides information about your outliers. There was a paper by Zong et al., DEEP AUTOENCODING GAUSSIAN MIXTURE MODEL FOR UNSUPERVISED ANOMALY DETECTION from the NEC labs at ICLR 2018 that used this approach. Please see https://openreview.net/forum?id=BJJLHbb0-

I have no idea how good that would work for your images. In the paper they used very small datasets.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.