1
$\begingroup$

I passages of text to classify by topic. I am using scikit learn, e.g. linear svc, but open to other options. Currently, use only the text of each passage (column labeled "En" below). But I feel that using the title of each passage (column labeled "Ref" below) would help.

    Ref               En                                                Topics
3   Gittin            modifier meaning board referred unique name mo... dinei-haget
11  Even HaEzer       hand katafres explanation hand slanted obvious... dinei-haget
67  Rest on Holiday   similar two baskets untithed fruit front first... laws-of-holidays
118 Beitzah           mishna states one ate food prepared festival e... laws-of-holidays
131 Sabbath           one may mix water salt oil dip one bread put c... rabbinically-forbidden-activities-on-shabbat

One excellent idea, which @Erwan mentioned in the comments, is to simply include the title together with the text of the passage. But I have two issues with that:

  1. How do I do that?
  2. Don't I want to make the title (or "Ref") count for more weight than the other words in the passage?
Improve this question
$\endgroup$
5
  • $\begingroup$ Well, why not concatenating the two columns for instance? $\endgroup$
    – Erwan
    Jun 16, 2020 at 14:21
  • $\begingroup$ @Erwan great! how would that help... you mean the title words would now just be part of the string of passage words? or is it more sophisticated than that? $\endgroup$
    – yishairasowsky
    Jun 16, 2020 at 14:34
  • $\begingroup$ Yes based on your question, the most simple option would be to use the whole text from title+article. If you want something more precise please update your question in order to provide more detail: which kind of classification algorithm you are using, what are the features etc. $\endgroup$
    – Erwan
    Jun 16, 2020 at 15:27
  • $\begingroup$ What are the exact features that you give to the classifier? I assume that you tokenize the text, probably one-hot encoding and maybe TFIDF? in case you give the whole column "En" as a single feature it's not going to work! $\endgroup$
    – Erwan
    Jun 17, 2020 at 17:29
  • $\begingroup$ @Erwan yes, i use tfidf etc. do i need to do anything else to make the weight of the words in the Ref column count for more? thanks! $\endgroup$
    – yishairasowsky
    Jun 18, 2020 at 7:50

1 Answer 1

Reset to default
2
$\begingroup$

In order to make the model take into account the two columns and distinguish whether a word is from the title or the article, you can generate the two sets of features independently and then concatenate the two vectors of features. This way the learning algorithm can assign different weights to a particular word depending on whether it comes from the title or the article.

However there is a disadvantage in doing that: depending on how many instances, how many words etc. there are, increasing the number of features might make the task harder for the model (for instance cause overfitting). It's probably worth testing the two options:

  • concatenate the text from the two columns then generate the features, i.e. no distinction between the columns but simpler job for the classifier
  • generate the features independently then concatenate the two sets of features (as above).
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.