Fine tuning Convolutional Neural Network with a learnable first layer

Question

I have a classification task using grayscale images and I want to leverage from pretrained networks.

There are a lot of resources out there presenting how to fine tune large neural nets like resnet, alexnet, etc for our custom task (usually with less data). However, I stumble on the problem that I want to use Resnet learned features on data which is not RGB (3 channels). In fact, I'm using grayscale images. I thought about using an "embedder" as a first layer before resnet which can transform my height x width x 1 image to a 224 x 224 x 3 resnet like input.

I wanted to know if there are works presenting this as an working approach (I couldn't get the right terminology to search for it). If so, is there a generic code to do so ?

class NeuralNet(nn.Module):
    def __init__(self,n_output):
        super().__init__()
        repr_size = 1024
        self.embedder = # The magic convolution code here that transform a 1 channel input to a 224 * 224 * 3 output
        self.backbone = resnet50
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(repr_size,128),
            nn.ReLU(),
            nn.Linear(128,n_output),
            # nn.Softmax(),
        )

Answer 1

It's impossible to fine-tune ResNet with grayscale images directly because it was trained with color images from ImageNet.

There are 5 solutions:

• Convert grayscale images to RGB images staying gray, but the results would be poor because ImageNet is not based on grayscale images.

  import cv2 #OpenCV

  backtorgb = cv2.cvtColor(gray,cv2.COLOR_GRAY2RGB)

• Color grayscale images using OpenCV and then use ResNet. It depends on the data you have. If the coloring works well, the results should be good with ResNet.

• Train Resnet (or any other CNN-based NN) from scratch using only your data with one gray channel. Therefore, you should have enough images.

class ResNet(nn.Module):

def __init__(self, block, layers, num_classes=1000):
    self.inplanes = 64
    super(ResNet, self).__init__()
    self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3,
                           bias=False)
    self.bn1 = nn.BatchNorm2d(64)
    self.relu = nn.ReLU(inplace=True)
    self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
    self.layer1 = self._make_layer(block, 64, layers[0])
    self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
    self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
    self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
    self.avgpool = nn.AvgPool2d(7, stride=1)
    self.fc = nn.Linear(512 * block.expansion, num_classes)
    self.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3,bias=False)

• Transform ImageNet to grayscale images and train ResNet with them and your data. This solution is the safest one but requires more training time.

• Modify ResNet with a Conv1 layer. Probably the smartest solution.