1
$\begingroup$

I'm trying to create something to help solve this problem:

My goal is that, given two images where one is an edit of the other, produce a system that outputs which one is most likely to be the original.

I have attempted to solve the problem in the following way.

  1. Collect a data set of random images that one may find in their social media blog. (10,000 images)

  2. Apply random lossy transformations to the image: resizing (with various scaling algorithms), cropping, lossy compression (JPEG)

  3. Preprocess input into training data:

    • Metadata:

      1. Collect file size, resolution, file format, lossy compression quality

      2. Add jitter (to prevent overfitting by recognizing specific images), jitter varies per 8x8 sample

      3. Output is a fixed number of floats per datum

    • Input pixels:

      1. Break the image into 8x8 chunks (the size is chosen to coincide with JPEG MCU size)

      2. Select a fixed number of chunks with the most entropy (changes in pixels)

      3. Select a color channel

      4. Divide by 255.0

      5. Output is 64 floats (one per pixel)

    • DCT values:

      1. Using the pixel data from above, perform a DCT transform

      2. Take the absolute value of the DCT value

      3. Output is 64 floats

  4. Feed the above data into a neural network.

    I don't know if it makes sense or not, but I started with a straight-forward model and did a simple hill-climbing hypersearch which added/removed/edited layers, and it produced the following model:

    model

    Each sample (and associated metadata) is fed as a single input.

  5. Fit the model to the labels 0.0 or 1.0, with 0.0 representing original (less noise) and 1.0 representing edited (more noise).

    Even though these labels don't describe any quality of the image itself, the goal is to use them to indicate some kind of ordering. The idea here is to "pull" the weights towards 0 for images known to have less noise and towards 1 for those with more noise, which I hope will cause the model to output higher values for images which have been edited.

Currently the model's accuracy is 88% (i.e. it correctly produces higher values for our edited versions for 88% of input image pairs).

My question is if there is a better way to represent ordering in this case other than labeling pairs 0 and 1. One potential issue with this is that the model is penalized if it outputs a score >1 or <0 (though currently avoided using sigmoid activation if that makes sense). Also, if there are other obvious improvements in the whole process. Thank you!

$\endgroup$
3
  • $\begingroup$ 88% accuracy is indeed very good. Although one should be careful of overfitting $\endgroup$
    – Nikos M.
    Mar 20 at 19:30
  • $\begingroup$ Thank you. I use 75% of samples for fitting and 25% for evaluation. The dropout layers seem to be working. $\endgroup$ Mar 20 at 19:45
  • $\begingroup$ If you use some thresholding on the $[0,1]$ output in order to classify images, you can fine-tune the threshold $\endgroup$
    – Nikos M.
    Mar 21 at 10:29
0
$\begingroup$

My question is if there is a better way to represent ordering in this case other than labeling pairs 0 and 1.

I solved this by creating an artificial comparator layer, which takes two inputs, and outputs 0 if the first is smaller or 1 if the first is greater. This allows backpropagation to correctly push the two away from each other within the same batch.

Also, if there are other obvious improvements in the whole process.

See my other question: Producing a confidence output to use in a weighted average layer

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.