0
$\begingroup$

I am using this pytorch script to learn and understand neural style transfer. I understood most part of the code but having some hard time understanding some parts of the code.

In line 15 Its not clear to me how model_activations work. I made a sample style tensor of the shape style.shape -> torch.Size([3, 300, 374]) and tried this sample code first without layers dict.

x = style
x = x.unsqueeze(0)
for name,layer in model._modules.items():
    x = layer(x)
    print(x.shape)

Output:

torch.Size([1, 64, 300, 374])
torch.Size([1, 64, 300, 374])
torch.Size([1, 64, 300, 374])
torch.Size([1, 64, 300, 374])
torch.Size([1, 64, 150, 187])
torch.Size([1, 128, 150, 187])
torch.Size([1, 128, 150, 187])
torch.Size([1, 128, 150, 187])
torch.Size([1, 128, 150, 187])
torch.Size([1, 128, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 75, 93])
torch.Size([1, 256, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 18, 23])
torch.Size([1, 512, 9, 11])

and then with layers

layers = {
    '0' : 'conv1_1',
    '5' : 'conv2_1',
    '10': 'conv3_1',
    '19': 'conv4_1',
    '21': 'conv4_2',
    '28': 'conv5_1'
    }

x = style
x = x.unsqueeze(0)
for name,layer in model._modules.items():
    if name in layers:
      x = layer(x)
      print(x.shape)

Output:
torch.Size([1, 64, 300, 374])
torch.Size([1, 128, 300, 374])
torch.Size([1, 256, 300, 374])
torch.Size([1, 512, 300, 374])
torch.Size([1, 512, 300, 374])
torch.Size([1, 512, 300, 374])
  1. My question is how the second method maintained the height and widght of style tensor (300, 374)?

The second confusion is in the line 91 where optimizer was set as optimizer = torch.optim.Adam([target],lr=0.007). In most pytorch tutorials I have seen them doing some thing like this optimizer = torch.optim.Adam(model.parameters(), lr = 0.01)

  1. Why is the optimizer initialization in neural style transfer different from other neural network tutorials?
  2. What is reason behind this optimizer = torch.optim.Adam([target],lr=0.007) >
$\endgroup$
0
1
$\begingroup$

Question 1

My question is how the second method maintained the height and widght of style tensor (300, 374)?

Applying the second piece of code does not call all layers but only the ones listed in the layers dict. In contrast to that the first method runs through all layers. Now, if you look at the network architecture:

IN: model
OUT:
Sequential(
  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): ReLU(inplace=True)
  (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (3): ReLU(inplace=True)
  (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (6): ReLU(inplace=True)
  (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (8): ReLU(inplace=True)
  (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (11): ReLU(inplace=True)
  (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (13): ReLU(inplace=True)
  (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (15): ReLU(inplace=True)
  (16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (17): ReLU(inplace=True)
  (18): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (19): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (20): ReLU(inplace=True)
  (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (22): ReLU(inplace=True)
  (23): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (24): ReLU(inplace=True)
  (25): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (26): ReLU(inplace=True)
  (27): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (29): ReLU(inplace=True)
  (30): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (31): ReLU(inplace=True)
  (32): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (33): ReLU(inplace=True)
  (34): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (35): ReLU(inplace=True)
  (36): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
)

You can see that in fact all max pooling layers are being skipped with the second method. Accordingly, your input does not get pooled, i.e. height and width remain unchanged. (the CNN layers in this network do not change height and width but only the channels)

If you slightly change the second piece of code to:

x = style
x = x.unsqueeze(0)
for name,layer in model._modules.items():
    x = layer(x)
    if name in layers:
        print(x.shape)

It runs through all layers again and gives:

torch.Size([1, 64, 300, 374])
torch.Size([1, 128, 150, 187])
torch.Size([1, 256, 75, 93])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 18, 23])

Question 2

Why is the optimizer initialization in neural style transfer different from other neural network tutorials?

In image style transfer learning you do not optimize the weights (model parameters) but the target image. Since you would like the target image to resemble your content and style pictures as close as possible. All weights (model parameters) are being kept constant. Therefore, the optimizer needs to receive the target image as an input and provide updates for it (and not for your model parameters like usually).


Question 3

What is reason behind this optimizer = torch.optim.Adam([target],lr=0.007) >

See Question 2. The target is initialized in line 69. Typically, you use the content image as a start:

target = content.clone().requires_grad_(True).to(device)

As you can see it also activates the gradient for the target. And this target image is then being optimized in order to minimize the loss total_loss = content_wt*content_loss + style_wt*style_loss.


A good read on the general approach is Image Style Transfer Using Convolutional Neural Networks.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.