# Finding parameters of image filter using classified pairs

I want to solve the problem of finding a parameter vector for an image filter (let us assume we know nothing about how the filter works, but we can feed it an input image and a set of parameters to produce an output image).

Thus, having a set $\{{I_k, J_k:=F_{\alpha}(I_k)}\}_{k\in\overline{1,N}}$ of $I_k$ images together with their filtered counterparts, $J_k$, what solutions would you recommend for finding $\alpha^\ast$ such that given $I^\ast$ the result $F_{\alpha^\ast}(I^\ast)$ is in the same "style" as the one of the $N$ training correspondence pairs.

I suppose one option is to use a convnet to transform $I_k$ into a feature vector, $v_k$, and then concatenate $\alpha_k$ to obtain $u_k =(v_k,\alpha_k)$. Once this is done, use a regression method to estimate the $\alpha^\ast$ part of $u^\ast$.

I would like to find an alternative solution to what seems like a candidate for the style transfer approach (e.g. https://arxiv.org/pdf/1703.07511.pdf). That approach seems to solve the problem differently, and I envision situations where I need to simply use a filter rather than let a network "guess the style of that filter".

Given the invoked no free lunch prospects, let us assume, for a paeticular problem from this class, that $F$ is a non-linear kernel-based filter that maps $I$ to $J$ as a result of an iterative and convergent process. More specifically, let $F$ be a mean shift filter with the $\alpha=(\rho, \sigma_s,\sigma_r)$ using a concatenated Gaussian kernel and a Parzen window of size $\rho$. Intuitively, I would be tempted to guess that this filter is not smooth w.r.t. $\alpha$, but a formal investigation is required (I suspect it is not smooth given that infinitesimal changes in the size of the window could shift the output towards another mode, indicating a step function behaviour).

In general, it is correct to assume that $\alpha \in \mathbb{R}^d$, with $d \ll N$.

Given the goal of finding $\alpha$ when both the filter action is known (either via numerical computation in general, or, in closed-form if the filter is a gaussian blur, for example), we can be confident that the $N$ input samples have non-constant $\alpha_k$ vector values to start with.

But for sake of generalizability, it would be more elegant to pursue a solution that does not need to know how the filter operates without actually applying it to an input. The first approach suggested in the comments and based on convnets seems to fit this scenario and the optimization problem is taking into account the filter error. However, it would be interesting to hear more opinions, perhaps involving shallow approaches, even at the expense of designing the solution to address the concrete mean shift filter example from above.

• When you describe the convent approach to estimate $\alpha_{k}$ do you you use $F$ at all ? You could compose $F$ with the convent $v_{k}$ and train the system fully end to end using $J_{k}$ and $a_{k}$ to construct your cost function. This way it is more explicit that you are trying to learn the filters parameters from both its target output and prior used filter params. – Francisco Vargas May 25 '17 at 13:33
• I am not using any posterior knowledge involving $F$ directly in the described approach/idea. I do not have a sufficient number of classified data to train a convnet from ground-up, so I would have to use one priorly trained to have it extract features. Would it help if I build the feature vectors as $(v_k, v'_k, \alpha_k)$, where $v'_k$ is the convnet feature vector corresponding to $J_k$? – teodron May 25 '17 at 14:03
• I am not sure it would help much. Doing the regression decoupled from the error that $F$ generates when using $\alpha_{k}$ is probably what would be the best to improve. How much data do you have ? – Francisco Vargas May 25 '17 at 15:41
• Currently I have not more than 200 pairs selected by manual parameter tuning of the $\alpha$ vector. So your suggestion would be to find the parameter vector via regression without plugging the feature vectors of the filtered images as additional dimensions of the sample space (that extra info should already be encoded in the $\alpha_k$ of the training samples. – teodron May 25 '17 at 16:06
• My initial suggestion was $\hat{\mathbf{W}} = \underset{\mathbf{W}} {arg \ min}\left \{ \sum_{k=1}^{N}(J_{k} - F(\sigma(I_{k}, \mathbf{W}), I_{k} ) )^{2} + (\alpha_{k} - \sigma(I_{k}, \mathbf{W}) )^{2} \right \}$ where $\sigma(I,\mathbf{W})$ is a convnet with weights $\mathbf{W}$. this way the convnet provides you with the parameter vector ($a^{*} = \sigma(I^{*} , \hat{\mathbf{W}})$ ). If you do not have enough data given the data is images, you can look into subtle transformations to augment your dataset. Salt and pepper, rotations, reflections , zca-whitening etc. – Francisco Vargas May 25 '17 at 16:22

Your parameter $\alpha$ has fairly low dimension. Therefore, I recommend that you apply optimization methods directly to try to find the best $\alpha$ (without trying to use convolutional neural networks and regression for this purpose).

Define a distance measure on images, $\|I-J\|$, to represent how dissimilar images $I,J$ are. You might use the squared $L_2$ norm for this, for instance.

Now, the loss for a particular parameter choice $\alpha$ is

$$L(\alpha) = \sum_{k=1}^N \|F_\alpha(I_k)-J_k\|.$$

We can now formulate your problem as follows: given a training set of images $(I_k,J_k)$, find the parameter $\alpha$ that minimizes the loss $L(\alpha)$.

A reasonable approach is to use some optimization procedure to solve this problem. You might use stochastic gradient descent, for instance. Because there might be multiple local minima, I would suggest that you start multiple instances of gradient descent from different starting points: use a grid search over the starting point. Since your $\alpha$ is only three-dimensional, it's not difficult to do a grid search over this three-dimensional space and then start gradient descent from each point in the grid. Stochastic gradient descent will allow you to deal with fairly large values of $N$.

This does require you to be able to compute gradients for $L(\alpha)$. Depending on the filter $\alpha$, it might be possible to symbolically calculate the gradients (perhaps with the help of a framework for this, such as Tensorflow); if that's too hard, you can use black-box methods to estimate the gradient by evaluating $L(\cdot)$ at multiple points.

If $L_2$ distance doesn't capture similarity in your domain, you could consider other distance measures as well.

I expect this is likely to be a more promising approach than what you sketched in the question, using convolutional networks and a regression model. (For one thing, there's no reason to expect the mapping from "features of $I_k$" to "features of $J_k$" to be linear, so there's no reason to expect linear regression to be effective here.)

• Its not necessary to compute the gradients for the filter. Look into Bayesian optimisation if your filter is indeed a blackbox. – Francisco Vargas May 28 '17 at 12:17
• Yes, computing the gradient of an iterative process analytically eludes me. I also do not think that $F_{\alpha}$ is smooth (w.r.t $\alpha$) and is definitely not guaranteed to be if $F$ is a black box. – teodron May 29 '17 at 6:26
• The drawback of seeking $\alpha$ given the $I_k$ and $J_k$ correspondences makes this formulation behave like a one size fits-all approach. Establishing feature correspondences between $I_k$, $\alpha_k$ and $J_k$ is definitely not guaranteed to be linear, but I am not fully convinced that, even with simpler filters, it is correct to assume the same $\alpha$ parameter value. In general, I would tend to regard the filter parameters as a function of the input. – teodron May 29 '17 at 6:30
• @teodron, you don't do it analytically; you do it in a blackbox way, e.g., $\nabla_h L(x)$ is approximated by $(L(x+ \epsilon h)-L(x))/\epsilon$ where $\epsilon$ is sufficiently small. I don't know what you mean by "one-size-fits-all". If you have a separate $\alpha$ value for each $I_k,J_k$ then you don't have a single problem; you have $N$ independent problems, one per value of $k$. If the filter parameters are a function of the input, then let $\alpha$ represent that function, and we're back to a single $\alpha$ for all the images. – D.W. May 29 '17 at 6:33
• I can think of a stupid example where the filter is a rotation plus translation of a black and white shape. Imagine that such a filter simply counts the number of corners of the shape, $N_c$. Then it translates the shape by $r*(cos(N_c\pi/12),sin(N_c\pi/12))$. Yes, the filter is stupid, but using a single value of the filter for any subsequent inputs would produce the same translation of the input regardless the number of corners. – teodron May 29 '17 at 6:36