Extract relevant vocabulary from a document

Question

I am training a DSSM model for QnA. I have 200 queries and their correspondent answers - the answer is answering what kind of information should an article related to the query contained. E.g:

Title: literacy rates Africa

Description: What are literacy rates in African countries

I have trained my model with the whole vocabulary but validation has no lead to great results. By the whole vocabulary I mean a list containing all the words used as I thought that eliminating prepositions, conjuctions and so on may lead to a loss in semantic meaning.

Now I am trying to find a way to extract the more relevant vocabulary of my documents. I have done some research and I have thought of n-grams. In fact there is a similar example in CNKT where the vocabulary they use for the answers is formed both for single words and n-grams but I couldn't find the way to do it by myself yet.

So far I have found a way to do the n-grams but this is not what I want as for example in the sentence:

the cow jumps over the moon

I get the following code:

the_cow_jumps
cow_jumps_over
jumps_over_the
over_the_moon

When I would be interested in the 4-gram:

cow_jumps_over_moon

Keep in mind that even eliminating the articles (or stopwords) I would still be getting more than one n-gram which is not what I want as my main objective is get the final vocabulary to train my model.

As an example of what I want could be the words:

book 
book_character 
book_editions_published 
book_subject 
books_published 

Instead of:

book
character
editions
published
subject

Answer 1

Not sure if this is what you are looking for. Couldn't understand your question properly do you want 4-grams then this is the way you could do it. Assuming you have cleaned your text file by eliminating articles.

from nltk import ngrams
file=open('abc.txt','r')
txt = file.read()
n = 4
fourgrams = ngrams(txt.split(), n)
for grams in fourgrams:
  print (grams)

In case your file abc.txt contains "This is random text to demonstrate the use of n-grams"

OUTPUT:
('This', 'is', 'random', 'text')
('is', 'random', 'text', 'to')
('random', 'text', 'to', 'demonstrate')
('text', 'to', 'demonstrate', 'the')
('to', 'demonstrate', 'the', 'use')
('demonstrate', 'the', 'use', 'of')
('the', 'use', 'of', 'n-grams')

If you do not want repetition then

words = 'This is random text we’re going to split apart'
x=[]
for word in words.split():
    x.append(word)
    if len(x) == 4:
        print(x)
        x=[]
print(x)

OUTPUT:
['This', 'is', 'random', 'text']
['we’re', 'going', 'to', 'split']
['apart']