Suppose I apply tri-gram indexing for my document collection, and is implementing a vector-space model to help retrieving the document. In the text it is mentioned implementing a trigram will introduce a new step in filtering the result. However, what are the problems that I need to be aware of if I implement tfidf/vector-space model? The reason I am exploring this option is to try handling basic spelling error handling, does it really work in practice?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
9
Trigram models can be more powerful for document retrieval than unigram models, but if you want to handle spelling errors, they will not be of much help. You need some form of fuzzy matching for that.
For example the string, "I like dosg too" would fool a unigram model because "dosg" is likely "dogs" misspelled, and it will encode it as "dosg" : 1. But you have the same problem in a trigram model. It will encode "I like dosg" : 1, "like dosg too" : 1. Which is not really better, as it will still not match any trigrams with the word "dogs" in it.
-
$\begingroup$ argh, i meant character n-gram, for instance "I like dosg" becomes
[" I ", " li", "lik", "ike", "ke ", " do", "dos", "osg", "sg "]$\endgroup$ Nov 9, 2015 at 1:49 -
$\begingroup$ That makes sense. That is a good feature space, though the best solution might be to use a default dictionary and map unknown (i.e. misspelled) terms to the nearest (edit distance) valid term. Of course that isn't always perfect. $\endgroup$– jamesmfNov 9, 2015 at 4:48
-
$\begingroup$ ooooo, yea, I am trying this character 3-gram with my prototype, the result looks interesting, but it is a lot slower than using unigram word model (: $\endgroup$ Nov 9, 2015 at 5:28
-
$\begingroup$ That would be the advantage of defining a dictionary first - you could use a unigram model with the preprocessing step of evaluating which tokens are a small distance away from an existing term. Most NLP packages include a dictionary. $\endgroup$– jamesmfNov 9, 2015 at 13:13
-
$\begingroup$ hmm, i don't see one for scikit-learn unfortunately $\endgroup$ Nov 11, 2015 at 1:46