I used 4 classifiers for my text data: NB, kNN, DT and SVM. As for NB and kNN I fully understand how they work with text - how we can count probabilities for all words in NB and how to use similarity metrics with TF-IDF vectors in kNN I don't understand at all how decision tree and support vector machine work with text data. I implemented all algorithms in Python so all I need is some resource or explanation how the other two classifiers work with text...

I understand DT with non-text data - it seams logical for example nodes with checking if some data is more/less than some number. But with text I get confused. Does it operate on text or with numerical vectors? The same applies to SVM...


1 Answer 1


Similarly to NB or kNN, the DT and SVM algorithms work with the features which are provided as input. So whenever ML is applied to text it's important to understand how the unstructured text is transformed into structured data, i.e. how text instances are represented with features.

There are many options, but traditionally a document is represented as as a vector over the full vocabulary. A very simple version of this is a boolean vector: a cell $v_i$ contains 1 if the word $w_i$ occurs in the document and 0 otherwise. The DT training will generate the tree the usual way, so in this case the conditions at the nodes will be v_i == 1, representing whether the word $w_i$ is present or not. If the values in the vector are say TFIDF weights, the conditions might look like v_i > 3.5 for instance. Similarly for SVM: the algorithm will find the optimal way to separate the instances in a multi-dimensional space: each dimension actually represents a single word, but the algorithm itself doesn't know (and doesn't care) about that.

  • $\begingroup$ so in case of SVM te multidimensional space will be in size of the max size of vecor from document (for example received by TFIDF method)? $\endgroup$ Jun 29, 2020 at 11:13
  • $\begingroup$ @jake-ferguson yes but it's not really the max size, it's the full size of the vocabulary across all the documents. All the documents must be represented with the same features, so their vectors can't be built independently (same for all four methods). $\endgroup$
    – Erwan
    Jun 29, 2020 at 11:39
  • $\begingroup$ One more @Erwan sorry. Can I find any example of building SVM classifier for text classification (long documents) from scratch? Let's say "on paper". Or it gets so complicated and multidimensional that it must be done in Python or any other language? $\endgroup$ Jul 1, 2020 at 10:50
  • 1
    $\begingroup$ @jake-ferguson it would certainly be too long to do it by hand on a really long document indeed. Anyway I don't know of any detailed example even with a short toy example, sorry. $\endgroup$
    – Erwan
    Jul 1, 2020 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.