Doc2vec '-' symbol occurrence

Question

Currently working on resume parser and struggled with embedding words with '-' symbols in them. Such as 'IT-manager'.

Vector representations of these words are incorrectly classified by doc2vec.

['it-manager'] [('salary', 0.23328335583209991), ('responsibilites', 0.22327110171318054), ('schedule', 0.14869527518749237), ('position', 0.12755176424980164)]

But when I remove '-' symbol, it is tokenized and classified right.

['it', 'manager'] [('position', 0.9306046962738037), ('schedule', 0.6630333662033081), ('responsibilites', 0.6081600189208984), ('salary', 0.5934453010559082)]

How do you work with such data properly? For this kind of task, I guess, it is better to exclude the symbol. But there may be a way to tell Doc2vec to treat these words like two different ones. Or perhaps tell the word_tokenizer to tokenize them in this fashion?

Answer 1

gensim's Phrases module may also be helpful:

from gensim.models import Phrases
documents = [
    "the mayor of new york was there",
    "machine learning can be useful sometimes",
    "new york mayor was present"
]

sentence_stream = [doc.split(" ") for doc in documents]

bigram = Phrases(sentence_stream, min_count=1, threshold=2)

sent = [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']
print(bigram[sent])
# Expected output:
# [u'the', u'mayor', u'of', u'new_york', u'was', u'there']

That code is from this other answer (I've copy-pasted it above for convenience).

For more on the Phrases module, check this page out.

Answer 2

Typically you would want to remove any symbols that do not contribute to the meaning of a token. In the case of 'it-manager' by removing the - you do not affect the interpretation of the word negatively. I would suggest filtering your vocabulary to identify all the words with other symbols in and make a judgement on whether you can filter the symbols without affecting the interpretation of the word.

You could do this using a regex filter such as:

m = re.search(r'[^\w]', <some string>)