I'm trying to understand and reproduce the CLIPascene paper. The paper is about the unsupervised generation of sketches from images using the expressive power of a CLIP classifier:
The idea seems to be the following (ignoring the simp part):
We start with some random input Z_init encoding a set of strokes. And additionally, for every generation, we start with a randomly initialised MLP_loc. Next, we calculate Z = Z_init + MLP_loc(Z_init). This Z should now be a better approximation of what strokes we need to receive a similar image. We put Z through a differentiable rasteriser and encode the resulting image using CLIP. This encoding can now be compared to a CLIP encoding of our target image. And we can use this comparison to make MLP_loc better.
This can be repeated until our sketch is good enough. After which we can just throw away MLP_loc, since it's only task was to generate this specific image.
And I have trouble understanding the point of this MLP_loc. What's the difference between this approach and just using gradient descent directly on the Z_init, skipping the MLP entirely?
I tried this latter approach and it always seems to get stuck in some local minimum that's not even close to a recognizable sketch. But for me it's still counterintuitive how the MLP solves this problem. Now I have to not only find a minimum of this CLIP error, but also find MLP weights that can take any noise vector and convert it into something close to this minimum vector.
Which in my eyes is twice the problem.