Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Has anybody seen any application that would use GAN that would take input image and would output image of the same size, that could be used as a layer for the first image. The layer would contain in eg. point of interest in the input image. Would that even be a good practice of the GAN usage?
I am looking for articles, examples of application that used something similar.
In terms of generating an image "layer", that is just the same as generating an output image that can be overlaid on the input using standard graphics software. If you want pixel-level accuracy in the output then the output will need to be the same size as the input, otherwise it could be smaller, provided it is the same aspect ratio, and in which case it would need to be scaled up in order to be used as an overlay. In any case, as the output of a GAN can be an image (and often is), then this part is easy.
The "G" in GAN stands for Generative. The purpose of a generative network is to create samples from a population, where there are typically many possibilities. Those samples can be conditioned on some additional data, and that additional data could be an image, although many examples will simpler conditioning, such as a category that the training output is representative of.
One possibility for using a GAN, is where your population contains a range of traits, and you can calculate vectors that control that trait. So you can take an input image, reconstruct it in the GAN then modify it by adding/subtracting the trait-related vector. An interesting example of this is Face Aging With Conditional Generative Adversarial Networks, and similar examples are around of adding/removing glasses etc. For this to work for you, you would literally need images that had your points-of interest in them and ones without them, and then you would be able to control addition/removal of points-of-interest. The network would not detect these points in the input, instead it would add them into the output. From reading your question this does not seem to be what you want.
A similar paper uses a GAN to remove rain from photos, based on training many images with and without rain then learning the "rain vector", encoding new images with rain in them into the GAN's internal representation and subtracting this "rain vector".
GANs conditioned on input images (as opposed to categories or internal embeddings) are also possible - this example of image completion might be closer to your goal. If your points of interest are variable with many options feasible, then it could work for you.
However, if your points of interest are supposed to always be the same pixels in each image, then your goal might be better defined by strict ground truth and become more like semantic segmentation, which can be attempted with variations on CNNs, such as described in this paper by Microsoft. These are much easier to set up and train than GANs, so if you can reasonably frame your problem as pixel classification from the original image, this is probably the way to go.
