1
$\begingroup$

I am training a DSSM model for QnA. I have 200 queries and their correspondent answers - the answer is answering what kind of information should an article related to the query contained. E.g:

Title: literacy rates Africa

Description: What are literacy rates in African countries

I have trained my model with the whole vocabulary but validation has no lead to great results. By the whole vocabulary I mean a list containing all the words used as I thought that eliminating prepositions, conjuctions and so on may lead to a loss in semantic meaning.

Now I am trying to find a way to extract the more relevant vocabulary of my documents. I have done some research and I have thought of n-grams. In fact there is a similar example in CNKT (https://cntk.ai/pythondocs/CNTK_303_Deep_Structured_Semantic_Modeling_with_LSTM_Networks.html) where the vocabulary they use for the answers is formed both for single words and n-grams but I couldn't find the way to do it by myself yet.

So far I have found a way to do the n-grams but this is not what I want as for example in the sentence:

the cow jumps over the moon

I get the following code:

the_cow_jumps
cow_jumps_over
jumps_over_the
over_the_moon

When I would be interested in the 4-gram:

cow_jumps_over_moon

Keep in mind that even eliminating the articles (or stopwords) I would still be getting more than one n-gram which is not what I want as my main objective is get the final vocabulary to train my model.

As an example of what I want could be the words:

book 
book_character 
book_editions_published 
book_subject 
books_published 

Instead of:

book
character
editions
published
subject
$\endgroup$
1
$\begingroup$

Not sure if this is what you are looking for. Couldn't understand your question properly do you want 4-grams then this is the way you could do it. Assuming you have cleaned your text file by eliminating articles.

from nltk import ngrams
file=open('abc.txt','r')
txt = file.read()
n = 4
fourgrams = ngrams(txt.split(), n)
for grams in fourgrams:
  print (grams)

In case your file abc.txt contains "This is random text to demonstrate the use of n-grams"

OUTPUT:
('This', 'is', 'random', 'text')
('is', 'random', 'text', 'to')
('random', 'text', 'to', 'demonstrate')
('text', 'to', 'demonstrate', 'the')
('to', 'demonstrate', 'the', 'use')
('demonstrate', 'the', 'use', 'of')
('the', 'use', 'of', 'n-grams')

If you do not want repetition then

words = 'This is random text we’re going to split apart'
x=[]
for word in words.split():
    x.append(word)
    if len(x) == 4:
        print(x)
        x=[]
print(x)

OUTPUT:
['This', 'is', 'random', 'text']
['we’re', 'going', 'to', 'split']
['apart']
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.