Currently working on resume parser and struggled with embedding words with '-' symbols in them. Such as 'IT-manager'.
Vector representations of these words are incorrectly classified by doc2vec.
['it-manager'] [('salary', 0.23328335583209991), ('responsibilites', 0.22327110171318054), ('schedule', 0.14869527518749237), ('position', 0.12755176424980164)]
But when I remove '-' symbol, it is tokenized and classified right.
['it', 'manager'] [('position', 0.9306046962738037), ('schedule', 0.6630333662033081), ('responsibilites', 0.6081600189208984), ('salary', 0.5934453010559082)]
How do you work with such data properly? For this kind of task, I guess, it is better to exclude the symbol. But there may be a way to tell Doc2vec to treat these words like two different ones. Or perhaps tell the word_tokenizer to tokenize them in this fashion?