2
$\begingroup$

In the paper Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network by Christian Ledig et al., the distance between images (used in the loss function) is calculated from feature maps $\phi_{i,j}$ extracted from the VGG19 network,
where $\phi_{i,j}$ is defined as "feature map obtained by the j-th convolution (after activation) before the i-th maxpooling layer".

Can you elaborate on how to calculate this feature map, may be for VGG54 mentioned in the paper?

$\phi_{5,4}$ means 4th convolutional layer before 5th max-pooling layer right? But 4th layer has so 512 filters. So we would have 512 feature spaces. Which one to choose from this? Also what does "after activation" mean?

I found this answer related to the same issue, but the answer didn't explain much.

Improve this question
$\endgroup$
2
$\begingroup$

In section 2.2.1 of the paper, they state that they use euclidean distance. I'm going to take your word that there are 512 filter activations in that layer; if I'm reading this right, there aren't 512 feature spaces, there is a 512-dimensional feature space that they are calculating euclidean distance in. So your distance function between two images $p$ and $q$ is just the standard Euclidean distance formula:

$$ d(\mathbf{p},\mathbf{q}) = \sqrt{\sum_{i=1}^{512}(p_i - q_i)^2}$$

where $\mathbf{p}$ and $\mathbf{q}$ are vectors holding the corresponding filter activations of $p$ and $q$.

Edit: Above the horizontal rule is my original answer which is wrong (or incomplete). What I think is happening is that the authors are taking the euclidean distance as above for each position in the feature maps at the $i,j$ layer, and averaging those distances to generate a scalar loss value. So for a 7x7 feature map, they'd be taking 49 512-dimensional euclidean distances and averaging them to get the VGG19 5,4 loss. This is how I read equation (5) in section 2.2.1 in their paper. I think the missing piece is that the authors don't bother with the square root in the euclidean distance formula. As discussed below, I think the notation is unclear.

Improve this answer
$\endgroup$
  • $\begingroup$ There are 512 filters. Each filter outputs a matrix right? So, there will be 512 matrices for Ground Truth Image and 512 matrices for Super Resolved Image. Right? So, p and q are not 1D vectors, instead they are matrices of size 7x7x512. How to calculate error here? qr.ae/TUhqpL Here is a overview of VGG19 ConvNet $\endgroup$ – Nagabhushan S N Nov 20 '18 at 19:01
  • $\begingroup$ You're right! The output of the convolutional layer is 7x7x512. I should have read the paper more thoroughly. But I think that the answer isn't much different. It looks like instead of 512-dimensional vectors, they're calculating the euclidean distance over all the entries in the 7x7x512. $\endgroup$ – Matthew Nov 20 '18 at 19:24
  • 1
    $\begingroup$ I've come back to this again and I think I'm wrong again, and that you're right to be confused. The description of the calculation doesn't seem to match the equation. They call it the euclidean distance between the feature representations, but equation (5) is more like a mean squared error. They sum over W and H, which are either 7 and 7 or 14 and 14, but not down the length-512 axis. This leaves the two phi terms as length-512 vectors but the notation doesn't make sense for that. $\endgroup$ – Matthew Nov 20 '18 at 19:49
  • 1
    $\begingroup$ Exactly! That's where I'm stuck. Of course, I can take average for 512 matrices, or cherry pick one of them, but that won't be what is suggested in the paper know... $\endgroup$ – Nagabhushan S N Nov 20 '18 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.