I am working on resume parsing script. I am trying to tag documents sentences with TaggedDocument function, provided by gensim.
What I have managed for now is to divide every text into sentence, put into one flat array and give every sentence an i (its order, basically) tag.
tagged_data = [TaggedDocument(words=word_tokenize(_d.lower()), tags=[str(i)]) for i, _d in enumerate(texts_flat)]
For the reason of possible improvement I want to tag every sentence not only with its order but with the name/order of text it is from. For that, I have made a list of lists, where every text is a list and every text contain list of sentences. i.e
texts = [text1 = [sent1, sent2, ...], text2, text3 ...]
- How to iterate over this kind of document?
I came up with smth like
tagged_data = [TaggedDocument(words=word_tokenize(_d.lower()), tags=[str(i) + '.'+ str(j)]) for i, j, _d in enumerate(texts)]but i get ValueError
ValueError: not enough values to unpack (expected 3, got 2)
- Is it even going to do anything?