Currently working on resume parser and struggled with embedding words with '-' symbols in them. Such as 'IT-manager'.

Vector representations of these words are incorrectly classified by doc2vec.

['it-manager'] [('salary', 0.23328335583209991), ('responsibilites', 0.22327110171318054), ('schedule', 0.14869527518749237), ('position', 0.12755176424980164)]

But when I remove '-' symbol, it is tokenized and classified right.

['it', 'manager'] [('position', 0.9306046962738037), ('schedule', 0.6630333662033081), ('responsibilites', 0.6081600189208984), ('salary', 0.5934453010559082)]

How do you work with such data properly? For this kind of task, I guess, it is better to exclude the symbol. But there may be a way to tell Doc2vec to treat these words like two different ones. Or perhaps tell the word_tokenizer to tokenize them in this fashion?


2 Answers 2


gensim's Phrases module may also be helpful:

from gensim.models import Phrases
documents = [
    "the mayor of new york was there",
    "machine learning can be useful sometimes",
    "new york mayor was present"

sentence_stream = [doc.split(" ") for doc in documents]

bigram = Phrases(sentence_stream, min_count=1, threshold=2)

sent = [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']
# Expected output:
# [u'the', u'mayor', u'of', u'new_york', u'was', u'there']

That code is from this other answer (I've copy-pasted it above for convenience).

For more on the Phrases module, check this page out.


Typically you would want to remove any symbols that do not contribute to the meaning of a token. In the case of 'it-manager' by removing the - you do not affect the interpretation of the word negatively. I would suggest filtering your vocabulary to identify all the words with other symbols in and make a judgement on whether you can filter the symbols without affecting the interpretation of the word.

You could do this using a regex filter such as:

m = re.search(r'[^\w]', <some string>)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.