I am training a DSSM model for QnA. I have 200 queries and their correspondent answers - the answer is answering what kind of information should an article related to the query contained. E.g:
Title: literacy rates Africa
Description: What are literacy rates in African countries
I have trained my model with the whole vocabulary but validation has no lead to great results. By the whole vocabulary I mean a list containing all the words used as I thought that eliminating prepositions, conjuctions and so on may lead to a loss in semantic meaning.
Now I am trying to find a way to extract the more relevant vocabulary of my documents. I have done some research and I have thought of n-grams. In fact there is a similar example in CNKT where the vocabulary they use for the answers is formed both for single words and n-grams but I couldn't find the way to do it by myself yet.
So far I have found a way to do the n-grams but this is not what I want as for example in the sentence:
the cow jumps over the moon
I get the following code:
the_cow_jumps cow_jumps_over jumps_over_the over_the_moon
When I would be interested in the 4-gram:
Keep in mind that even eliminating the articles (or stopwords) I would still be getting more than one n-gram which is not what I want as my main objective is get the final vocabulary to train my model.
As an example of what I want could be the words:
book book_character book_editions_published book_subject books_published
book character editions published subject