1
$\begingroup$

I am currently working on a Binary Text Classification Model (False Information Detection) using Support Vector Machine and used TF-IDF as text vectorizer in Python. I have already tried training the model but upon testing, I have encountered a problem:

For example I have the model predicted an entry saying "COVID-19 is happening today" as "True", but after changing the text into "COVID-19 is not happening today", it is still predicted as "True", in which should be predicted as "False".

Where does the problem lie in this situation?

How can we make the algorithm classify text with opposite meanings like ones mentioned above?

Note:

  • The text that exists in the dataset I used in modelling is “COVID-19 is happening today.”

  • I used also predict_proba to know the probability of the text being 0(False), or 1(True). It shows that the two entries I made have the same output in predict_proba which with this I can say that it reads the two entries as the same (maybe as "COVID-19 is happening today").

$\endgroup$

1 Answer 1

1
$\begingroup$

You should probably lower your expectations about what a ML model can achieve.

  • First, it's a statistical process: the model just predicts the most likely label. Unavoidably, some errors are going to happen.
  • The model doesn't understand anything about the meaning, it just calculates the probability of the label based on the parameters learned during training.
  • Bag of words representation, like TFIDF, is a very simplified representation of the text meaning. With this representation, the model can only have "rules" such as: "if the instance contains word $x$, then the label is more likely to be True". This kind of representation deals decently with simple text classification tasks in which some specific groups of words are strongly associated with a label. This is not sufficient to properly handle complex linguistic constructions such as negations, metaphors, sarcasm, conditionals, ...
  • There are more complex text representations available, but to my knowledge none of them can achieve near perfect Natural Language Understanding, which would be needed for this task. In case you want to achieve a state of the art system for this task, you would have to study the state of the art in natural language inference.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.