I'm working on a ML.Net based feature to extract keywords from a document corpus using TD-IDF.

Given this test corpus (one document per line):

This is an example to compute n-grams.
N-gram is a sequence of 'N' consecutive words/tokens.
ML.NET's ProduceNgrams API produces vector of n-grams.
Each position in the vector corresponds to a particular n-gram.
The value at each position corresponds to,
the number of times n-gram occured in the data (Tf), or
the inverse of the number of documents that contain the n-gram (Idf),
or compute both and multiply together (Tf-Idf).

And this test document:

foobar This is an example to compute n-grams

I'm getting the following result:

example             = 2,0794
example|compute     = 2,0794
compute             = 1,3863
compute|n           = 0,0000
n                   = 0,0000
n|grams             = 0,0000
grams               = 0,0000
n|gram              = 0,0000
gram                = 0,0000
gram|sequence       = 0,0000

My question is about the term foobar which is not part of the training corpus, and therefore not part of the result set. Is there a recommended way of dealing with out-of-vocabulary terms when using TF-IDF?

  • $\begingroup$ Using BPE (Byte Pair encoding) would be a way to handle out-of-vocabulary items. You'd be using subwords instead of or as well as words. It's not a new technique but has recently been popularized in LLMs. $\endgroup$ Commented Mar 11 at 20:30

1 Answer 1


In my opinion, there is no way to deal out-of-vocabulary terms in TF-IDF as it only works using the test corpus. It's like you are trying to find certain word like technology in sports corpus.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.