0
$\begingroup$

I'm trying to assign a vector to every word in my sentence. Some words are not recognised even though very similar words are - for example: Going, gone, go are recognised while goes isn't.

How should I go about assigning any logical values to the word goes or any word of this like? Please notice that I don't know in advance what words won't be recognised.

$\endgroup$
  • $\begingroup$ In your example you can treat them the same by using the lemma. I suppose you can't use a pre-trained embedding to avoid this problem? $\endgroup$ – Emre Aug 29 '16 at 6:59
3
$\begingroup$

The words Going, Gone, Goes are considered to be similar in only one context i.e. they all have the same root word Go. This is known as Stemming/Lemmatization, both of which are special cases of normalization process. So, in order to assign some value to these words, you have to first lemmatize them. In order to get more details about this, I am attaching two links which can be helpful in your case:

http://textminingonline.com/dive-into-nltk-part-iv-stemming-and-lemmatization

https://stackoverflow.com/questions/771918/how-do-i-do-word-stemming-or-lemmatization

| improve this answer | |
$\endgroup$
1
$\begingroup$

Generally for words which are not found in the vocabulary, a zero vector is assigned to them.

Some other hacks which I could think of are :

  1. Try different morphologies of the word (ex. morphology of goes will give go and you can get vector of go to initialize goes).

  2. Try finding similar word from wordnet. If you are able to find embedding of similar word from wordnet, initialize original word's embedding with this word's embedding.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.