Is it a good idea to combine fine tuning and feature extraction techniques?

Question

I have a normal/tumor medical images dataset and, for the same patients, also the relative genomics, and my goal is to predict if a patient has a tumor by combining all the information.

To achieve this, I am using a ResNet50 with imagenet weights to extract features from images, and other methods to extract features from genes. I join the two features and use an SVM to make a prediction.

The accuracy isn't extremely bad, but I wanted to know if it can be increased by performing a fine-tuning over the same images, in place of using a network only trained on the imagenet datasets as it is.

I have researched papers but I found literally nothing, neither in favor nor in contrast with this hypothesis.

Is there some known contraindication?

Answer 1

The concept of multimodal learning is relevant: in this case, combining data from two modalities: 1) image signal using ResNet50 and 2) genomic features extracted from genes. Multimodal learning for extending state-of-the-art performance of pre-trained unimodal models is currently an area of active research in the literature. In the NLP domain, the paper on Multimodal Bert describes combining a pre-trained language model with additional signals from visual and auditory inputs for improved classification performance.

In the specific scenario mentioned here, combining the image and non-image signals into a single deep neural network graph would allow for additional fine-tuning. It is possible to first train the layer combining signals from both modalities, then later fine-tune all weights in the graph (with a lower learning rate) for improved performance. This procedure discussed a bit in the Keras Guide to Transfer Learning.