Hybrid Convolutional and Conventional Neural Networks

Question

Convolutional Neural Networks are usually the best choice for classification and semantic segmentation of images. categorical/numerical data (age, height, city, etc.) on the other hand are best processed by conventional machine learning models, such as (deep) Random Forests, Support Vector Machines or Conventional Neural Networks.

Are there hybrid architectures combining both convolutional and conventional Neural Networks for classification of datasets consisting of images and categorical data? I am sure this problem has been solved before and I am specifically looking for research papers, tutorials and implementations in one of the common libraries (PyTorch, Tensorflow, Keras, etc.).

A good example for such a "hybrid" dataset is the ISIC dataset, containing thousands of images of skin growths together with information such as the age and the sex of the patient for each image. Hypothetically, a hybrid model that includes this categorical/numerical information could be more succesful in detecting skin cancer, than a model which just uses the images.

Answer 1

If I had to do it, I would use a transfer learning strategy:

I would train a deep learning model just with the images to solve a classification problem. In order to do this, I would need of course to have tags for the image classes. If this tags do not exist, then this approach does not make sense. It is relatively easy to make very accurate image classification, as there are lots of tools. Of course, I would do that using a CNN.

Once I have my CNN trained, I would take the last fully connected layer out. I would concatenate the inputs of this layer with the cateogrical/numerical data that you have, and obtain a feature vector using this concatenation. I would input this feature vector to another ML algorithm to carry out the objective that you need.

Did I just make this up?

No. A similar thing is done in fast ai deep learning course for text. An RNN is trained to predict the next word of a sentence and then the weights of that RNN are used to build a sentiment analysis machine. I think the idea is very similar and this is why I call it transfer learning.

What if I don't have tags for the images?

Then my advise is to use a CNN trained on image-net, take the last fully connected layer out and concatenate the inputs to that layer with the other features.