I'm reading someone else's code, which uses vgg19 to transfer the style of an artist to a 'content' image. I'm wondering why they have put
INI_NOISE_RATIO * noise_img + (1. - INI_NOISE_RATIO) * content_img)
instead of just noise_img (INI_NOISE_RATIO=0.7 here).
Why use a mix of the noise image and content image as input for the white noise bit in the content loss? The paper https://arxiv.org/pdf/1508.06576.pdf by Gatys et.al doesn't mention the mixing?
The loss functions are not related to the noise image.
The network is run twice to determine parameters to the loss functions. Once for the content image, and once for the style image. The code you are asking about occurs afterwards.
First the network is run against the content image (note no noise mixing here):
sess.run([net['input'].assign(content_img)])
Aside: Uses an assignment function - not a feed_dict - because in order for the style transfer to work, the input has become a tf variable, not a placeholder.
Then the first loss function is defined against some of the network parameters from this run (this extracts the activations of interest from the networks as fixed params to the loss function - because it uses output from sess.run, not references to tensor variables):
cost_content = sum(map(lambda l,: l[1]*
build_content_loss(sess.run(net[l[0]]),net[l[0]]) , CONTENT_LAYERS))
The same thing is done for the style image:
sess.run([net['input'].assign(style_img)])
cost_style = sum(map(lambda l: l[1]*
build_style_loss(sess.run(net[l[0]]), net[l[0]]), STYLE_LAYERS))
And a final cost function defined:
cost_total = cost_content + STYLE_STRENGTH * cost_style
Now, after the cost function is defined, the code initialises a new input (which is also our style transfer output):
sess.run(net['input'].assign( INI_NOISE_RATIO* noise_img +
(1.-INI_NOISE_RATIO) * content_img))
This line has nothing to do with defining loss. It just initialises the network ready to generate gradients against a starting image and the loss functions that were already defined beforehand. You can start with pure noise or a copy of the content image to start the process, and each will give a slightly different result.
The original paper started from a random image. Quoting from the paper (on page 10):
Thus we can change the initially random image $\tilde{x}$ until it generates the same response in a certain layer of the CNN [. . .]
. . . however, starting from the content image was tried and used very soon afterwards by many in practice, and it works well.
This mix of both approaches is what the author has chosen, presumably because it gets pleasing results. As there are no lower layers in the network than the input, in fact you can start from an arbitrary image (e.g. all zeros), and you should still get a result, although it may not look as nice.
To (sort of) answer your question as written:
Why use a mix of the noise image and content image as input for the white noise bit in the content loss?
The code is not doing that. The mixing is occurring as you say. But the line you show is not part of the content loss. The mixed image is the initial solution, and iterations will turn it into the stylised image.
