12
$\begingroup$

I have understood how GAN works while two networks (generative and discriminative) compete with each other. I have built a DCGAN (GAN with convolutional discriminator and de-convolutional generator) which now successfully generates handwritten digits similar to those in MNIST dataset.

I have read a lot about GAN's applications for extracting features from images. How can use my trained GAN model (on MNIST dataset) to extract feature from MNIST handwritten digist images?

$\endgroup$
  • $\begingroup$ For feature extraction, I am getting the feature size of 128*120(i.e. 64+32+16+8)=15360. I am sure that I am missing something. I have another question regarding choosing features vector. Should I consider (conv2d+batchnorm+activation) weights or only conv2d weights during feature extraction? $\endgroup$ – Tanmoy Dam Jan 23 '19 at 22:16
5
$\begingroup$

Typically to extract features, you can use the top layer of the network before the output. The intuition is that these features are linearly separable because the top layer is just a logistic regression.

For GANs, you can use the features from the discriminator. These features are supposed to give a probability if the input came from the training dataset, "real images". In Radford's DCGAN paper, they use all the convolutional layers of the discriminator and run a max pooling layer extract features for CIFAR-10.

To evaluate the quality of the representations learned by DCGANs for supervised tasks, we train on Imagenet-1k and then use the discriminator’s convolutional features from all layers, maxpooling each layers representation to produce a 4 × 4 spatial grid. These features are then flattened and concatenated to form a 28672 dimensional vector and a regularized linear L2-SVM classifier is trained on top of them.

| improve this answer | |
$\endgroup$
2
$\begingroup$

Kenny's answer is correct – if you're using convolutional D, output of layers before dense may serve as features. My intuition is that it will work better for AC-GANs (or similar architectures, which make D classify input in addition to determining if it's fake or real).

There is an approach called BiGAN which adds an Encoder component able to map generated and training samples to latent distribution z used to "initialize" generator. Authors show that it can effectively be used as a feature set for transfer learning and other tasks.

| improve this answer | |
$\endgroup$
1
$\begingroup$

as the GAN consists of two parts -- the generator and the discriminator, there are two ways to use GAN as feature extractor:

  1. Generator based way as presented by Mikhail Yurasov.
  2. Discriminator based way as presented by Kenny.

The second way is more controversial. Some studies [1] thought that, intuitively, as the target of the discriminator is to distinguish the generated samples from the real samples, it will just focus on the difference between these two kinds of samples. But what makes sense is the difference between real samples, which is the samples used by the downstream tasks.

I have tried to study this, and I found that the feature extracted can be factorized into two orthogonal subspaces. The first space contributes to the discriminator task, while the second is free from it. As in most cases, the features used to distinguish the real samples from the generated ones are noise, the second feature space will be noise-free. From this perspective, although the task of discriminator will not focus on the difference between real samples, which are useful for the downstream tasks, the noise-free features contained in the second subspace will work.

[1] Jost Tobias Springenberg. Unsupervised and Semi-Supervised Learning with Categorical Generative Adversarial Networks. arXiv:1511.06390 [cs, stat], April 2016. arXiv preprint. arXiv:1511.06390 [stat.ML]. Ithaca, NY: Cornell University Library.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.