Suppose I am interested in classifying a set of instances composed by different content types, e.g.:

  • a piece of text
  • an image

as relevant or non-relevant for a specific class C.

In my classification process I perform the following steps:

  1. Given a sample, I subdivide it in text and image
  2. A first SVM binary classifier (SVM-text), trained only on text, classifies the text as relevant/non-relevant for the class C
  3. A second SVM binary classifier (SVM-image), trained only on images, classifies the image as relevant/non-relevant for the class C

Both SVM-text and SVM-image produce an estimate of the probability of the analyzed content (text or image) of being relevant for the class C. Given this, I am able to state whether the text is relevant for C and the image is relevant for C.

However, these estimates are valid for segments of the original sample (either the text or the image), while it is not clear how to obtain a general opinion on the whole original sample (text+image). How can I combine conveniently the opinions of the two classifiers, so as to obtain a classification for the whole original sample?


1 Answer 1


Basically, you can do one of two things:

  1. Combine features from both classifiers. I.e., instead of SVM-text and SVM-image you may train single SVM that uses both - textual and visual features.
  2. Use ensemble learning. If you already have probabilities from separate classifiers, you can simply use them as weights and compute weighted average. For more sophisticated cases there are Bayesian combiners (each classifier has its prior), boosting algorithms (e.g. see AdaBoost) and others.

Note, that ensembles where initially created for combining different learners, not different sets if features. In this later case ensembles have advantage mostly in cases when different kinds of features just can't be combined in a single vector efficiently. But in general, combing features is simpler and more straightforward.

  • $\begingroup$ I have read something about ensemble learning (see scientific paper ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1105916&tag=1). However, it is not clear to me how to apply them. SVM-text is returning a probability of belonging to class C, and the same holds for SVM-image. Is it sufficient to average them? You talk about weighted average: how to select weights? $\endgroup$
    – Eleanore
    Commented Sep 16, 2014 at 13:48
  • $\begingroup$ There's huge number of ways to incorporate weights. In simplest case of binary classification you can use equal weights for both answers (from image- and text-based classifiers) and thus simply average probabilities. For multinomial classification you can treat probabilities themselves as weights and compute probability for each class separately. Or you can use adaptive weights like in AdaBoost. Or anything else. Just start reading about weighted ensembles and choose algorithm that in your opinion is the most appropriate for the task. $\endgroup$
    – ffriend
    Commented Sep 16, 2014 at 20:04
  • $\begingroup$ Averaging is often not the best approach. I would recommend reading mlwave.com/kaggle-ensembling-guide $\endgroup$
    – pir
    Commented Oct 23, 2015 at 12:20
  • $\begingroup$ It might be interesting to extract features from both modalities and apply a single SVM so it would be able to find interactions between text and image features. $\endgroup$
    – pir
    Commented Oct 23, 2015 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.