I'm working on a ML.Net based feature to extract keywords from a document corpus using TD-IDF.
Given this test corpus (one document per line):
This is an example to compute n-grams. N-gram is a sequence of 'N' consecutive words/tokens. ML.NET's ProduceNgrams API produces vector of n-grams. Each position in the vector corresponds to a particular n-gram. The value at each position corresponds to, the number of times n-gram occured in the data (Tf), or the inverse of the number of documents that contain the n-gram (Idf), or compute both and multiply together (Tf-Idf).
And this test document:
foobar This is an example to compute n-grams
I'm getting the following result:
example = 2,0794 example|compute = 2,0794 compute = 1,3863 compute|n = 0,0000 n = 0,0000 n|grams = 0,0000 grams = 0,0000 n|gram = 0,0000 gram = 0,0000 gram|sequence = 0,0000
My question is about the term
foobar which is not part of the training corpus, and therefore not part of the result set. Is there a recommended way of dealing with out-of-vocabulary terms when using TF-IDF?