1
$\begingroup$

I have a classification task using grayscale images and I want to leverage from pretrained networks.

There are a lot of resources out there presenting how to fine tune large neural nets like resnet, alexnet, etc for our custom task (usually with less data). However, I stumble on the problem that I want to use Resnet learned features on data which is not RGB (3 channels). In fact, I'm using grayscale images. I thought about using an "embedder" as a first layer before resnet which can transform my height x width x 1 image to a 224 x 224 x 3 resnet like input.

I wanted to know if there are works presenting this as an working approach (I couldn't get the right terminology to search for it). If so, is there a generic code to do so ?

class NeuralNet(nn.Module):
    def __init__(self,n_output):
        super().__init__()
        repr_size = 1024
        self.embedder = # The magic convolution code here that transform a 1 channel input to a 224 * 224 * 3 output
        self.backbone = resnet50
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(repr_size,128),
            nn.ReLU(),
            nn.Linear(128,n_output),
            # nn.Softmax(),
        )

$\endgroup$
1
  • $\begingroup$ The "embedder" would be analogous to an image colorization network. As far as I know, image colorization is far from being solved in the general case. $\endgroup$
    – noe
    Jul 21 at 22:28

1 Answer 1

2
$\begingroup$

It's impossible to fine-tune ResNet with grayscale images directly because it was trained with color images from ImageNet.

There are 5 solutions:

  • Convert grayscale images to RGB images staying gray, but the results would be poor because ImageNet is not based on grayscale images.

    import cv2 #OpenCV

    backtorgb = cv2.cvtColor(gray,cv2.COLOR_GRAY2RGB)

  • Color grayscale images using OpenCV and then use ResNet. It depends on the data you have. If the coloring works well, the results should be good with ResNet.

  • Train Resnet (or any other CNN-based NN) from scratch using only your data with one gray channel. Therefore, you should have enough images.

class ResNet(nn.Module):

def __init__(self, block, layers, num_classes=1000):
    self.inplanes = 64
    super(ResNet, self).__init__()
    self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3,
                           bias=False)
    self.bn1 = nn.BatchNorm2d(64)
    self.relu = nn.ReLU(inplace=True)
    self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
    self.layer1 = self._make_layer(block, 64, layers[0])
    self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
    self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
    self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
    self.avgpool = nn.AvgPool2d(7, stride=1)
    self.fc = nn.Linear(512 * block.expansion, num_classes)
    self.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3,bias=False)
  • Transform ImageNet to grayscale images and train ResNet with them and your data. This solution is the safest one but requires more training time.

  • Modify ResNet with a Conv1 layer. Probably the smartest solution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.