4
$\begingroup$

I have set of images which they look like the below sample.

enter image description here

You can think of it as a person's 2-week schedule represented by different color. Inside the images. However, there are some small missing areas shown in black. I'm interested in inpaint/fill the missing areas using ML methods.

Note that I have already tried classical methods like some those mentioned in the paper: "Image Inpainting, Christine Guillemot and Olivier Le Meur", but the result was not satisfactory.

Since almost all images have the same pattern, I'd like to know if anyone can suggest the ML-based approaches tackle this problem.

Improve this question
$\endgroup$
2
  • $\begingroup$ Many ML approaches learn from data, so it's relevant to know whether you have complete images (i.e. images without missing data) and how many you have (because some ML approaches need a lot of data). For real images, you may use deep learning (e.g. this), but such an approach seems overkill to me for your problem. Also, why are you approaching the problem as image gap filling instead of e.g. multilabel classification? $\endgroup$
    – noe
    Jun 7, 2017 at 8:45
  • 1
    $\begingroup$ Thanks, @ncasas. I do have so many complete images without missing data as well. Regarding why using the image gap filling method, is that we tried some other methods. Now try to look the problem from image processing perspective and compare the result. But specifically for multilabel classification, do you happen any paper addressing the similar problem? $\endgroup$
    – ARASH
    Jun 8, 2017 at 2:22

2 Answers 2

Reset to default
3
$\begingroup$

If I were you I would use deep learning. You can use an autoencoder format for this. Essentially you would feed the image in, then each layer yields a smaller output. Then you would feed the output in reverse, using transposed weight matrices, which would yield progressively larger outputs. The final output from the backward pass would be the filled in image. Since your image has such a small color range I would recommend representing pixel colors using one-hot encoding.

Improve this answer
$\endgroup$
1
$\begingroup$

It looks like there is special structure in your images that might allow you to do better than using generic image inpainting methods.

It's hard to know from just one example image, but it is possible that you might be able to build a satisfactory method using only very simple features of the input image. In particular, you could train a classifier that, given a black pixel, tries to predict the correct color for that pixel.

In your specific example, some features that look useful are: "the most common color in the same column", "the most common color in the same row", "the color of the closest non-black pixel above your pixel (in the same column)", "the color of the closest non-black pixel below your pixel (in the same column)", "the color of the closest non-black pixel to the right of your pixel (in the same row)", "the color of the closest non-black pixel to the left of your pixel (in the same row)". I suggest trying to train a simple classifer on a few simple features like this, and see how well it works.

If you want to get fancy, you could try modelling this as a Markov random field over a grid graph. In other words, each pixel is a vertex in the graph, and two pixels are connected by an edge if they are adjacent. There are then standard methods to learn the underlying density functions given some training examples, and once you know the density functions, you could then use maximum likelihood methods to infer the values of the black pixels.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.