I am wondering how to convert caffe reference model trained with ImageNet (color pics) for grayscale image to save memory and to speed up.

The filters in the caffe convolution layers are different for RGB but this can be, for example, averaged to form one channel filter.

Since my data is grayscale and speed at test (prediction) is important for me, this may help.

I can of course convert my grayscale images in RGB representation to reuse pretrained convolution layers as they are. But I want to avoid it if possible.

My questions are:

Would this be possible?

How exactly one should combine filters? (Average?)

After filters converted, weights between conv layers need to be re-trained?

  • $\begingroup$ I think it is an interesting question, but the general case is likely possible but the performance will degrade. Also converting greyscale to RGB would also be worse in a similar way. In both cases this is because the colour channel information is being used by the filters separately. In specific cases, the impact of this could be small enough you could get away with it . . . as usual the only way to be sure is to try. I'd suggest trying with grayscale images converted to RGB first to get a sense of whether the accuracy loss is acceptable. $\endgroup$ – Neil Slater Jul 11 '16 at 14:42
  • $\begingroup$ Would you then recommend to use pre-train model for MNIST digit (grayscale) instead? But digit data would not represent more complex shapes... And can you elaborate the last part about color information? $\endgroup$ – otterb Jul 11 '16 at 14:49
  • $\begingroup$ oh, i did not expand your comment. Ok, so maybe I will just try with averaging and see what I get. Thanks! $\endgroup$ – otterb Jul 11 '16 at 14:52
  • $\begingroup$ sir i am interesting about your question . does it possible to convert filters from 3 channels int one instead convert our data images into RGB thanks $\endgroup$ – Sam Ab Nov 19 '17 at 6:16

I doubt either of your proposed approaches will work, these convolutional layers learn all kind of relationships between the colors which will be non-existent in grayscale, upsampling it to RGB makes your input space very different from the input space that ImageNet was trained on and averaging these filters of the trained ImageNet model makes a whole lot of assumptions that are unlikely to hold. The second approach does seem more promising than the first, so if you want to test an approach I would go for the second one. It's possible that it is at least a better initialization than randomly. I think you will end up needing a lot of data and computation anyway, so it might be better to start off with your own network shape and start from scratch.

  • $\begingroup$ Thanks for your intuition. I feel that at least it's not a totally bad idea to try. And comments from you and Neil really help me guess how much time I should invest. I will go with 2nd option and see. $\endgroup$ – otterb Jul 13 '16 at 10:51

What about pre training your own greyscale Imagenet model? Create a single channel resnet architecture and train it on images that have already been converted to greyscale. Training a model from scratch on Imagenet may seem like a daunting task but it can be done quite quickly and cheaply these days.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.