Suppose I apply tri-gram indexing for my document collection, and is implementing a vector-space model to help retrieving the document. In the text it is mentioned implementing a trigram will introduce a new step in filtering the result. However, what are the problems that I need to be aware of if I implement tfidf/vector-space model? The reason I am exploring this option is to try handling basic spelling error handling, does it really work in practice?


Trigram models can be more powerful for document retrieval than unigram models, but if you want to handle spelling errors, they will not be of much help. You need some form of fuzzy matching for that.

For example the string, "I like dosg too" would fool a unigram model because "dosg" is likely "dogs" misspelled, and it will encode it as "dosg" : 1. But you have the same problem in a trigram model. It will encode "I like dosg" : 1, "like dosg too" : 1. Which is not really better, as it will still not match any trigrams with the word "dogs" in it.

  • $\begingroup$ argh, i meant character n-gram, for instance "I like dosg" becomes [" I ", " li", "lik", "ike", "ke ", " do", "dos", "osg", "sg "] $\endgroup$ – Jeffrey04 Nov 9 '15 at 1:49
  • $\begingroup$ That makes sense. That is a good feature space, though the best solution might be to use a default dictionary and map unknown (i.e. misspelled) terms to the nearest (edit distance) valid term. Of course that isn't always perfect. $\endgroup$ – jamesmf Nov 9 '15 at 4:48
  • $\begingroup$ ooooo, yea, I am trying this character 3-gram with my prototype, the result looks interesting, but it is a lot slower than using unigram word model (: $\endgroup$ – Jeffrey04 Nov 9 '15 at 5:28
  • $\begingroup$ That would be the advantage of defining a dictionary first - you could use a unigram model with the preprocessing step of evaluating which tokens are a small distance away from an existing term. Most NLP packages include a dictionary. $\endgroup$ – jamesmf Nov 9 '15 at 13:13
  • $\begingroup$ hmm, i don't see one for scikit-learn unfortunately $\endgroup$ – Jeffrey04 Nov 11 '15 at 1:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.