I am working on resume parsing script. I am trying to tag documents sentences with TaggedDocument function, provided by gensim.

What I have managed for now is to divide every text into sentence, put into one flat array and give every sentence an i (its order, basically) tag.

tagged_data = [TaggedDocument(words=word_tokenize(_d.lower()), tags=[str(i)]) for i, _d in enumerate(texts_flat)]

For the reason of possible improvement I want to tag every sentence not only with its order but with the name/order of text it is from. For that, I have made a list of lists, where every text is a list and every text contain list of sentences. i.e

texts = [text1 = [sent1, sent2, ...], text2, text3 ...]
  1. How to iterate over this kind of document? I came up with smth like tagged_data = [TaggedDocument(words=word_tokenize(_d.lower()), tags=[str(i) + '.'+ str(j)]) for i, j, _d in enumerate(texts)] but i get ValueError

ValueError: not enough values to unpack (expected 3, got 2)

  1. Is it even going to do anything?

I've managed to write a function that iterates over it for me:

class TaggedDocumentIterator(object):
def __init__(self, doc_list, labels_list):
    self.labels_list = labels_list
    self.doc_list = doc_list
def __iter__(self):
    for idx, doc in enumerate(self.doc_list):
        for idy in enumerate(doc):
            yield TaggedDocument(words=idy[1].split(), tags=[str(self.labels_list[idx]) + '.' + str(doc.index(idy[1]))])

docLabels = list(range(len(texts))) data = list(texts) sentences = TaggedDocumentIterator(data, docLabels)

| improve this answer | |
  • $\begingroup$ out of curiosity (if you're still around), how did this model tagging system go? $\endgroup$ – frei Feb 18 '19 at 8:06
  • $\begingroup$ @frei still working on this project. Still using this function :) This model was pretty good for tagging the resume into blocks, containing some specific info: education, experience, skills etc. Now I moved a bit further, and for each of these blocks I have a model for classification. So, for education block, there are University class, period of education class, specialization class etc. $\endgroup$ – Graygood Feb 18 '19 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.