# Combine multiple classifiers to build a multi-modal classifier

Suppose I am interested in classifying a set of instances composed by different content types, e.g.:

• a piece of text
• an image

as relevant or non-relevant for a specific class C.

In my classification process I perform the following steps:

1. Given a sample, I subdivide it in text and image
2. A first SVM binary classifier (SVM-text), trained only on text, classifies the text as relevant/non-relevant for the class C
3. A second SVM binary classifier (SVM-image), trained only on images, classifies the image as relevant/non-relevant for the class C

Both SVM-text and SVM-image produce an estimate of the probability of the analyzed content (text or image) of being relevant for the class C. Given this, I am able to state whether the text is relevant for C and the image is relevant for C.

However, these estimates are valid for segments of the original sample (either the text or the image), while it is not clear how to obtain a general opinion on the whole original sample (text+image). How can I combine conveniently the opinions of the two classifiers, so as to obtain a classification for the whole original sample?

1. Combine features from both classifiers. I.e., instead of SVM-text and SVM-image you may train single SVM that uses both - textual and visual features.
• I have read something about ensemble learning (see scientific paper ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1105916&tag=1). However, it is not clear to me how to apply them. SVM-text is returning a probability of belonging to class C, and the same holds for SVM-image. Is it sufficient to average them? You talk about weighted average: how to select weights? – Eleanore Sep 16 '14 at 13:48