# Fine tuning Convolutional Neural Network with a learnable first layer

I have a classification task using grayscale images and I want to leverage from pretrained networks.

There are a lot of resources out there presenting how to fine tune large neural nets like resnet, alexnet, etc for our custom task (usually with less data). However, I stumble on the problem that I want to use Resnet learned features on data which is not RGB (3 channels). In fact, I'm using grayscale images. I thought about using an "embedder" as a first layer before resnet which can transform my height x width x 1 image to a 224 x 224 x 3 resnet like input.

I wanted to know if there are works presenting this as an working approach (I couldn't get the right terminology to search for it). If so, is there a generic code to do so ?

class NeuralNet(nn.Module):
def __init__(self,n_output):
super().__init__()
repr_size = 1024
self.embedder = # The magic convolution code here that transform a 1 channel input to a 224 * 224 * 3 output
self.backbone = resnet50
self.classifier = nn.Sequential(
nn.Flatten(),
nn.Linear(repr_size,128),
nn.ReLU(),
nn.Linear(128,n_output),
# nn.Softmax(),
)


• The "embedder" would be analogous to an image colorization network. As far as I know, image colorization is far from being solved in the general case.
– noe
Jul 21 at 22:28

It's impossible to fine-tune ResNet with grayscale images directly because it was trained with color images from ImageNet.

There are 5 solutions:

• Convert grayscale images to RGB images staying gray, but the results would be poor because ImageNet is not based on grayscale images.

import cv2 #OpenCV

backtorgb = cv2.cvtColor(gray,cv2.COLOR_GRAY2RGB)

• Color grayscale images using OpenCV and then use ResNet. It depends on the data you have. If the coloring works well, the results should be good with ResNet.

• Train Resnet (or any other CNN-based NN) from scratch using only your data with one gray channel. Therefore, you should have enough images.

class ResNet(nn.Module):

def __init__(self, block, layers, num_classes=1000):
self.inplanes = 64
super(ResNet, self).__init__()
self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3,
bias=False)
self.bn1 = nn.BatchNorm2d(64)
self.relu = nn.ReLU(inplace=True)