I am trying to predict the probability of a disease by using audio and images, the audio and the images do not come from the same source. I am thinking of combining the outputs (maybe average them) of two models one utilizing image data to calculate the probability the other using audio data. I am feeling unsure of this approach as I will not be able to verify the final combined output. Will the outcomes be reliable, or should I use a different approach?


1 Answer 1


You can try to fuse the image and audio data. You can fuse in different ways:

1) Early fusion - fusion at feature level

2) Late fusion - fusion at output level (similar to your idea)

Here is some papers for your reference:

Look, Listen and Learn https://arxiv.org/pdf/1705.08168.pdf

Learn to Combine Modalities in Multimodal Deep Learning https://arxiv.org/pdf/1805.11730.pdf

  • $\begingroup$ Thanks for the answer, the problem is that I am using audio and the visual data from two different datasets and I will not be able evaluate the accuracy of the final output. $\endgroup$
    – ayushdubey
    Commented Jun 8, 2020 at 6:39
  • 1
    $\begingroup$ It would probably help if you could give some idea about the datasets and their labels. There are also papers on weak supervision if that is applicable. $\endgroup$
    – Anshul G.
    Commented Jun 15, 2020 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.