Finding parameters of image filter using classified pairs

Question

I want to solve the problem of finding a parameter vector for an image filter (let us assume we know nothing about how the filter works, but we can feed it an input image and a set of parameters to produce an output image).

Thus, having a set $\{{I_k, J_k:=F_{\alpha}(I_k)}\}_{k\in\overline{1,N}}$ of $I_k$ images together with their filtered counterparts, $J_k$, what solutions would you recommend for finding $\alpha^\ast$ such that given $I^\ast$ the result $F_{\alpha^\ast}(I^\ast)$ is in the same "style" as the one of the $N$ training correspondence pairs.

I suppose one option is to use a convnet to transform $I_k$ into a feature vector, $v_k$, and then concatenate $\alpha_k$ to obtain $u_k =(v_k,\alpha_k)$. Once this is done, use a regression method to estimate the $\alpha^\ast$ part of $u^\ast$.

I would like to find an alternative solution to what seems like a candidate for the style transfer approach (e.g. https://arxiv.org/pdf/1703.07511.pdf). That approach seems to solve the problem differently, and I envision situations where I need to simply use a filter rather than let a network "guess the style of that filter".

Additional details and possible assumptions

Given the invoked no free lunch prospects, let us assume, for a paeticular problem from this class, that $F$ is a non-linear kernel-based filter that maps $I$ to $J$ as a result of an iterative and convergent process. More specifically, let $F$ be a mean shift filter with the $\alpha=(\rho, \sigma_s,\sigma_r)$ using a concatenated Gaussian kernel and a Parzen window of size $\rho$. Intuitively, I would be tempted to guess that this filter is not smooth w.r.t. $\alpha$, but a formal investigation is required (I suspect it is not smooth given that infinitesimal changes in the size of the window could shift the output towards another mode, indicating a step function behaviour).

In general, it is correct to assume that $\alpha \in \mathbb{R}^d$, with $d \ll N$.

Given the goal of finding $\alpha$ when both the filter action is known (either via numerical computation in general, or, in closed-form if the filter is a gaussian blur, for example), we can be confident that the $N$ input samples have non-constant $\alpha_k$ vector values to start with.

But for sake of generalizability, it would be more elegant to pursue a solution that does not need to know how the filter operates without actually applying it to an input. The first approach suggested in the comments and based on convnets seems to fit this scenario and the optimization problem is taking into account the filter error. However, it would be interesting to hear more opinions, perhaps involving shallow approaches, even at the expense of designing the solution to address the concrete mean shift filter example from above.

Answer 1

Your parameter $\alpha$ has fairly low dimension. Therefore, I recommend that you apply optimization methods directly to try to find the best $\alpha$ (without trying to use convolutional neural networks and regression for this purpose).

Define a distance measure on images, $\|I-J\|$, to represent how dissimilar images $I,J$ are. You might use the squared $L_2$ norm for this, for instance.

Now, the loss for a particular parameter choice $\alpha$ is

$$L(\alpha) = \sum_{k=1}^N \|F_\alpha(I_k)-J_k\|.$$

We can now formulate your problem as follows: given a training set of images $(I_k,J_k)$, find the parameter $\alpha$ that minimizes the loss $L(\alpha)$.

A reasonable approach is to use some optimization procedure to solve this problem. You might use stochastic gradient descent, for instance. Because there might be multiple local minima, I would suggest that you start multiple instances of gradient descent from different starting points: use a grid search over the starting point. Since your $\alpha$ is only three-dimensional, it's not difficult to do a grid search over this three-dimensional space and then start gradient descent from each point in the grid. Stochastic gradient descent will allow you to deal with fairly large values of $N$.

This does require you to be able to compute gradients for $L(\alpha)$. Depending on the filter $\alpha$, it might be possible to symbolically calculate the gradients (perhaps with the help of a framework for this, such as Tensorflow); if that's too hard, you can use black-box methods to estimate the gradient by evaluating $L(\cdot)$ at multiple points.

If $L_2$ distance doesn't capture similarity in your domain, you could consider other distance measures as well.

I expect this is likely to be a more promising approach than what you sketched in the question, using convolutional networks and a regression model. (For one thing, there's no reason to expect the mapping from "features of $I_k$" to "features of $J_k$" to be linear, so there's no reason to expect linear regression to be effective here.)