I have a program that takes an image and a list of parameters and generates a new image. I would like to automate the selection of parameters to produce the 'best' image.
'Best' in this context doesn't have a formal definition, it's whatever the human user thinks it is - it may be how close it resembles the source image, or it could be some other artistic criterion. The user themselves can't necessarily tell what makes an image 'good', however they can compare two generated images and tell which one they prefer.
Question: What is the best suited optimization algorithm to select the parameters? The properties of the search as follows:
- User can't assign an absolute score to the output, but instead can only compare 2 images at a time and tell which image is better or whether they're about equal. (-1, 0, 1)
- Small changes in the parameters result in small changes in the generated image.
- User may judge small differences in the output inconsistently. We may make assumptions about the smallest change in the input that produces a perceptible change in the output. If necessary, it's also possible to ask the user to assign a score on how sure they are about their judgement for a given pair of images.
- Single maximum for the 'best' image is not guaranteed, but is likely. It's unlikely that there is a large number of local maxima.
- There are about 5 parameters, all continuous and bounded.
- We assume that the search starts from a selection of parameters that produce a decent but not necessarily optimal result.
- The evaluation cost is considered high, because it takes some time to generate and judge each pair.
- Because of the high evaluation cost, the process should ideally take less than 100 comparisons.
- It's not obvious to the user how the parameters affect the output, therefore we can't directly ask them for a hint in the right direction.