Clarification wanted for make_step function of Google's deep dream script

Question

From https://github.com/google/deepdream/blob/master/dream.ipynb

def objective_L2(dst):          # Our training objective. Google has since release a way to load
    dst.diff[:] = dst.data      # arbitrary objectives from other images. We'll go into this later.

def make_step(net, step_size=1.5, end='inception_4c/output', 
              jitter=32, clip=True, objective=objective_L2):
    '''Basic gradient ascent step.'''

    src = net.blobs['data'] # input image is stored in Net's 'data' blob
    dst = net.blobs[end]

    ox, oy = np.random.randint(-jitter, jitter+1, 2)
    src.data[0] = np.roll(np.roll(src.data[0], ox, -1), oy, -2) # apply jitter shift

    net.forward(end=end)
    objective(dst)  # specify the optimization objective
    net.backward(start=end)
    g = src.diff[0]
    # apply normalized ascent step to the input image
    src.data[:] += step_size/np.abs(g).mean() * g

    src.data[0] = np.roll(np.roll(src.data[0], -ox, -1), -oy, -2) # unshift image

    if clip:
        bias = net.transformer.mean['data']
        src.data[:] = np.clip(src.data, -bias, 255-bias)

If I understand what is going on correctly, the input image in net.blobs['data'] is inserted into the NN until the layer end. Once, the forward pass is complete until end, it calculates how "off" the blob at the end is from "something".

Questions

• What is this "something"? Is it dst.data? I stepped through a debugger and found that dst.data was just a matrix of zeros right after the assignment and then filled with values after the backward pass.

• Anyways, assuming it finds how "off" the result of the forward pass is, why does it try to do a backwards propagation? I thought the point of deep dream wasn't to further train the model but "morph" the input image into whatever the original model's layer represents.

• What exactly does src.data[:] += step_size/np.abs(g).mean() * g do? It seems like applying whatever calculation was done above to the original image. Is this line what actually "morphs" the image?

Links that I have already read through

• https://stackoverflow.com/a/31028871/2750819

I would be interested in what the author of the accepted answer meant by

we take the original layer blob and "enchance" signals in it. What does it mean, I don't know. Maybe they just multiply the values by coefficient, maybe something else.

• http://www.kpkaiser.com/machine-learning/diving-deeper-into-deep-dreams/

In this blog post the author comments next to src.data[:] += step_size/np.abs(g).mean() * g: "get closer to our target data." I'm not too clear what "target data" means here.

Note I'm cross posting this from https://stackoverflow.com/q/40690099/2750819 as I was recommended to in a comment.

Answer 1

What is this "something"? Is it dst.data? I stepped through a debugger and found that dst.data was just a matrix of zeros right after the assignment and then filled with values after the backward pass.

Yes. dst.data is the working contents of the layer inside the CNN that you are trying to maximise. The idea is that you want to generate an image that has a high neuron activation in this layer by making changes to the input. If I understand this correctly though, it should be populated immediately after the forward pass here: net.forward(end=end)

Anyways, assuming it finds how "off" the result of the forward pass is, why does it try to do a backwards propagation? I thought the point of deep dream wasn't to further train the model but "morph" the input image into whatever the original model's layer represents.

It is not training. However, like with training we cannot directly measure how "off" the source that we want to change is from an ideal value, so instead we calculate how to move toward a better value by taking gradients. Back propagation is the usual method for figuring out gradients to parameters in the CNN. There are some main differences with training:

• Instead of trying to minimise a cost, we want to increase a metric which summarises how excited the target layer is - we are not trying to find any stationary point (e.g. the maximum possible value), and instead Deep Dream usually just stops arbitrarily after a fixed number of iterations.

• We back propagate further than usual, all the way to the input layer. That's not something you do normally during training.

• We don't use any of the gradients to the weights. The weights in the neural network are never changed. We are only interested in the gradients at the input - but to get them we need to calculate all the others first.

What exactly does src.data[:] += step_size/np.abs(g).mean() * g do? It seems like applying whatever calculation was done above to the original image. Is this line what actually "morphs" the image?

It takes a single step in the image data along the gradients that we have calculated will trigger more activity in the target layer. Yes this alters the input image. It should be repeated a few times, according to how extreme you want the Deep Dream effect to be.