0
$\begingroup$

I am trying to predict the probability of a disease by using audio and images, the audio and the images do not come from the same source. I am thinking of combining the outputs (maybe average them) of two models one utilizing image data to calculate the probability the other using audio data. I am feeling unsure of this approach as I will not be able to verify the final combined output. Will the outcomes be reliable, or should I use a different approach?

$\endgroup$
0
$\begingroup$

You can try to fuse the image and audio data. You can fuse in different ways:

1) Early fusion - fusion at feature level

2) Late fusion - fusion at output level (similar to your idea)

Here is some papers for your reference:

Look, Listen and Learn https://arxiv.org/pdf/1705.08168.pdf

Learn to Combine Modalities in Multimodal Deep Learning https://arxiv.org/pdf/1805.11730.pdf

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the answer, the problem is that I am using audio and the visual data from two different datasets and I will not be able evaluate the accuracy of the final output. $\endgroup$ – ayushdubey Jun 8 at 6:39
  • $\begingroup$ It would probably help if you could give some idea about the datasets and their labels. There are also papers on weak supervision if that is applicable. $\endgroup$ – Anshul G. Jun 15 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.