Skip to main content
Data Science

Return to Question

Corrected spelling, grammar and added tag
Source Link
edit approved Mar 11, 2019 at 16:43
HFulcher
edit approved Mar 11, 2019 at 16:43
HFulcher
  • 182
  • 13

Currently working on resume parser and struggled with word embedding on wordwords with '-' symbols in itthem. Such as 'IT-manager'.

If I get vector representationVector representations of this word,these words are incorrectly classified by doc2vec fails to classify it right.

['it-manager'] [('salary', 0.23328335583209991), ('responsibilites', 0.22327110171318054), ('schedule', 0.14869527518749237), ('position', 0.12755176424980164)]

But when I remove '-' symbol, it is tokenized and classified right.

['it', 'manager'] [('position', 0.9306046962738037), ('schedule', 0.6630333662033081), ('responsibilites', 0.6081600189208984), ('salary', 0.5934453010559082)]

How todo you work with such data properly? For this kind of task, I guess, it is better to exclude the symbol. But there may be there is thea way to tell Doc2vec to threattreat these words like two different ones. Or toperhaps tell the word_tokenizer to tokenize them in this fashion?

Currently working on resume parser and struggled with word embedding on word with '-' symbols in it. Such as 'IT-manager'.

If I get vector representation of this word, doc2vec fails to classify it right.

['it-manager'] [('salary', 0.23328335583209991), ('responsibilites', 0.22327110171318054), ('schedule', 0.14869527518749237), ('position', 0.12755176424980164)]

But when I remove '-' symbol, it is tokenized and classified right.

['it', 'manager'] [('position', 0.9306046962738037), ('schedule', 0.6630333662033081), ('responsibilites', 0.6081600189208984), ('salary', 0.5934453010559082)]

How to work with such data properly? For this kind of task, I guess, it is better to exclude symbol. But may be there is the way to tell Doc2vec to threat these words like two different ones. Or to tell the word_tokenizer to tokenize them in this fashion?

Currently working on resume parser and struggled with embedding words with '-' symbols in them. Such as 'IT-manager'.

Vector representations of these words are incorrectly classified by doc2vec.

['it-manager'] [('salary', 0.23328335583209991), ('responsibilites', 0.22327110171318054), ('schedule', 0.14869527518749237), ('position', 0.12755176424980164)]

But when I remove '-' symbol, it is tokenized and classified right.

['it', 'manager'] [('position', 0.9306046962738037), ('schedule', 0.6630333662033081), ('responsibilites', 0.6081600189208984), ('salary', 0.5934453010559082)]

How do you work with such data properly? For this kind of task, I guess, it is better to exclude the symbol. But there may be a way to tell Doc2vec to treat these words like two different ones. Or perhaps tell the word_tokenizer to tokenize them in this fashion?

Source Link
Graygood
Graygood
  • 113
  • 5

Doc2vec '-' symbol occurrence

Currently working on resume parser and struggled with word embedding on word with '-' symbols in it. Such as 'IT-manager'.

If I get vector representation of this word, doc2vec fails to classify it right.

['it-manager'] [('salary', 0.23328335583209991), ('responsibilites', 0.22327110171318054), ('schedule', 0.14869527518749237), ('position', 0.12755176424980164)]

But when I remove '-' symbol, it is tokenized and classified right.

['it', 'manager'] [('position', 0.9306046962738037), ('schedule', 0.6630333662033081), ('responsibilites', 0.6081600189208984), ('salary', 0.5934453010559082)]

How to work with such data properly? For this kind of task, I guess, it is better to exclude symbol. But may be there is the way to tell Doc2vec to threat these words like two different ones. Or to tell the word_tokenizer to tokenize them in this fashion?