I have a normal/tumor medical images dataset and, for the same patients, also the relative genomics, and my goal is to predict if a patient has a tumor by combining all the information.

To achieve this, I am using a ResNet50 with imagenet weights to extract features from images, and other methods to extract features from genes. I join the two features and use an SVM to make a prediction.

The accuracy isn't extremely bad, but I wanted to know if it can be increased by performing a fine-tuning over the same images, in place of using a network only trained on the imagenet datasets as it is.

I have researched papers but I found literally nothing, neither in favor nor in contrast with this hypothesis.

Is there some known contraindication?


The concept of multimodal learning is relevant: in this case, combining data from two modalities: 1) image signal using ResNet50 and 2) genomic features extracted from genes. Multimodal learning for extending state-of-the-art performance of pre-trained unimodal models is currently an area of active research in the literature. In the NLP domain, the paper on Multimodal Bert describes combining a pre-trained language model with additional signals from visual and auditory inputs for improved classification performance.

In the specific scenario mentioned here, combining the image and non-image signals into a single deep neural network graph would allow for additional fine-tuning. It is possible to first train the layer combining signals from both modalities, then later fine-tune all weights in the graph (with a lower learning rate) for improved performance. This procedure discussed a bit in the Keras Guide to Transfer Learning.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.