4
$\begingroup$

I am working on a problem that current depends on word level embeddings created using Word2Vec. I am researching new methods to apply to this model and one was a character level embedding. I have not found much information on it, and I don't imagine Word2Vec but at a character level would be effective. Is there any insight on giving vector representations to characters for an overall classification model?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

It completely depends on what you're classifying.

Using character embeddings for semantic classification of sentences introduces unnecessary complexity, making the data harder to fit. Although using n-grams would help the model deal with word derivatives.

Classifying words based on their derivative would be a task that would require character embeddings.

If you're asking whether it would be useful to train a model to embed characters like you would with word2vec- then no. And in fact would probably yield bad results. We use embeddings to implicitly encode that two data points are close together and therefore should be treated more similar to the model. The letter 'd' shouldn't be semantically closer to 'e' than 'q'.

Improve this answer
$\endgroup$
4
  • $\begingroup$ That's a good point, I will look more into n-gram solutions thank you. $\endgroup$
    – Jacob B
    Commented Aug 24, 2018 at 16:58
  • 1
    $\begingroup$ I think it is inaccurate to say that it is not useful to pre-train character embeddings. colinmorris.github.io/blog/1b-words-char-embeddings illustrates how character embeddings can be useful. $\endgroup$
    – jkyh
    Commented Apr 12, 2019 at 3:46
  • $\begingroup$ @jkyh Then isn't is a good that I never said that? $\endgroup$
    – Daniel
    Commented Apr 15, 2019 at 11:08
  • 1
    $\begingroup$ Sorry, I might have misunderstood you. What do you mean by "If you're asking whether it would be useful to train a model to embed characters like you would with word2vec- then no."? $\endgroup$
    – jkyh
    Commented Apr 16, 2019 at 12:04
1
$\begingroup$

For the sake of completeness, fastText uses sub-word information (character level n-grams) when creating embeddings and then for classification with some interesting results:

  • can give prediction for the unknown words that have some similarity with known ones
  • with interesting results for modelling syntactic relations like 'went' - 'go' + 'give' = 'gave'
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.