Convolutional Neural Networks are usually the best choice for classification and semantic segmentation of images. categorical/numerical data (age, height, city, etc.) on the other hand are best processed by conventional machine learning models, such as (deep) Random Forests, Support Vector Machines or Conventional Neural Networks.

Are there hybrid architectures combining both convolutional and conventional Neural Networks for classification of datasets consisting of images and categorical data? I am sure this problem has been solved before and I am specifically looking for research papers, tutorials and implementations in one of the common libraries (PyTorch, Tensorflow, Keras, etc.).

A good example for such a "hybrid" dataset is the ISIC dataset, containing thousands of images of skin growths together with information such as the age and the sex of the patient for each image. Hypothetically, a hybrid model that includes this categorical/numerical information could be more succesful in detecting skin cancer, than a model which just uses the images.

  • $\begingroup$ I am not so sure about it being so widely solved. $\endgroup$ Jun 12, 2018 at 8:06
  • $\begingroup$ I seems to be a common problem, however - doesn't it? $\endgroup$ Jun 12, 2018 at 8:51
  • 1
    $\begingroup$ Maybe, but for sure it does not appear in standard deep learning courses. There is now a kaggle competition that deals with it, but I haven't seen it anywhere outside from that. $\endgroup$ Jun 12, 2018 at 8:53
  • $\begingroup$ Absolutely, this is advanced stuff - that's why I came here. See the updated question for an example of a "hybrid" dataset. $\endgroup$ Jun 12, 2018 at 8:55
  • 1
    $\begingroup$ I'll give my recommendation, although I am aware it is not exactly what you ask. I hope it helps, though $\endgroup$ Jun 12, 2018 at 9:00

1 Answer 1


If I had to do it, I would use a transfer learning strategy:

I would train a deep learning model just with the images to solve a classification problem. In order to do this, I would need of course to have tags for the image classes. If this tags do not exist, then this approach does not make sense. It is relatively easy to make very accurate image classification, as there are lots of tools. Of course, I would do that using a CNN.

Once I have my CNN trained, I would take the last fully connected layer out. I would concatenate the inputs of this layer with the cateogrical/numerical data that you have, and obtain a feature vector using this concatenation. I would input this feature vector to another ML algorithm to carry out the objective that you need.

Did I just make this up?

No. A similar thing is done in fast ai deep learning course for text. An RNN is trained to predict the next word of a sentence and then the weights of that RNN are used to build a sentiment analysis machine. I think the idea is very similar and this is why I call it transfer learning.

What if I don't have tags for the images?

Then my advise is to use a CNN trained on image-net, take the last fully connected layer out and concatenate the inputs to that layer with the other features.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.