I am working on a problem that current depends on word level embeddings created using Word2Vec. I am researching new methods to apply to this model and one was a character level embedding. I have not found much information on it, and I don't imagine Word2Vec but at a character level would be effective. Is there any insight on giving vector representations to characters for an overall classification model?


It completely depends on what you're classifying.

Using character embeddings for semantic classification of sentences introduces unnecessary complexity, making the data harder to fit. Although using n-grams would help the model deal with word derivatives.

Classifying words based on their derivative would be a task that would require character embeddings.

If you're asking whether it would be useful to train a model to embed characters like you would with word2vec- then no. And in fact would probably yield bad results. We use embeddings to implicitly encode that two data points are close together and therefore should be treated more similar to the model. The letter 'd' shouldn't be semantically closer to 'e' than 'q'.

| improve this answer | |
  • $\begingroup$ That's a good point, I will look more into n-gram solutions thank you. $\endgroup$ – Jacob B Aug 24 '18 at 16:58
  • $\begingroup$ I think it is inaccurate to say that it is not useful to pre-train character embeddings. colinmorris.github.io/blog/1b-words-char-embeddings illustrates how character embeddings can be useful. $\endgroup$ – jkyh Apr 12 '19 at 3:46
  • $\begingroup$ @jkyh Then isn't is a good that I never said that? $\endgroup$ – Daniel Apr 15 '19 at 11:08
  • 1
    $\begingroup$ Sorry, I might have misunderstood you. What do you mean by "If you're asking whether it would be useful to train a model to embed characters like you would with word2vec- then no."? $\endgroup$ – jkyh Apr 16 '19 at 12:04

For the sake of completeness, fastText uses sub-word information (character level n-grams) when creating embeddings and then for classification with some interesting results:

  • can give prediction for the unknown words that have some similarity with known ones
  • with interesting results for modelling syntactic relations like 'went' - 'go' + 'give' = 'gave'
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.