For purpose of quite big project I am doing a text mining on some documents. My steps are quite common:

  1. All to lower case
  2. Tokenization
  3. Stop list and stop words
  4. Lemmatizaton
  5. Stemming
  6. Some other steps like removing symbols.

Then I prepare bag of words, make DTF and classify to 3 classes with SVM and Naive Bayes.

But the accuracy I get is not too high (50-60%). I think that may be because in array of words after all the steps are still many words very irrelevant like first and last name from document. What is the attitude in such case? What can be done during preprocessing to make classifiers work better with higher accuracy?

I was thinking about preparing some dictionary consists of all relevant words to my area but it could be too hard and for sure some important words will be missed.

Any advice what could be done here?


1 Answer 1


The question is very broad, so the general answer is: it depends on the specifics of your data and the problem you're trying to solve. The general idea would be to analyze what's happening:

  • Is it actually possible to solve the problem given the indications in the data? Would a human expert do better than 50-60% accuracy given only the information in the data? If yes which clues would they use, and are these potential clues directly available as features to the learning algorithm?
  • Ratio between size of data and size of features: if there are not enough instances, the algorithm doesn't have enough to generalize from. If there are too many features, the algorithm is likely to overfit (i.e. consider things which happen by chance in the data as significant patterns). And of course if there are not enough features the algorithm doesn't have the indications necessary to make accurate predictions.

Common things to try:

  • remove features (here words) which appear very rarely, because they can't help and can cause overfitting.
  • if the problem seems to be that words are not informative enough, try with bigrams or even trigrams... but be careful to avoid overfitting, the number of features is likely to increase a lot.
  • use weights (e.g. TFIDF) to help the learning algorithm focus on the relevant features.
  • depending on the task, more sophisticated methods could be relevant: topic modeling, disambiguation techniques, syntax...
  • $\begingroup$ and is it OK to prepare a dictionary of words that are relevant to problem and then before classification keep just words from dictionary? Or we don't do this in text-mining and it is not a good approach? $\endgroup$ Commented Nov 23, 2019 at 14:00
  • $\begingroup$ Sure it's ok, the use of terminology resources is common in many specific domains. The only problem with that is that it often requires a lot of work to create it and maintain it. $\endgroup$
    – Erwan
    Commented Nov 23, 2019 at 14:15
  • $\begingroup$ and does it have any "professional" name ? Is it called dictionary like I use the term or maybe somehow different? $\endgroup$ Commented Nov 23, 2019 at 14:31
  • 1
    $\begingroup$ Imho the word "dictionary" is fine but it can make people think that there is a definition or something associated with each word. Technically if only words from this list are taken into account then it can be called the "vocabulary", but it's also common to just call this a "list of terms" (with variants: list of target/domain terms, etc.). $\endgroup$
    – Erwan
    Commented Nov 23, 2019 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.