8
$\begingroup$

I have been working on a trained data for Word2vec algorithm. Since we need words to stay as original we don't make them lowercase at the preprocessing phase. Thus there are words with different variations (e.g "Earth" and "earth").

The only way I can think of is to take average of vectors for "Earth" and "earth" to create a single vector to represent the word. (Since the feature vector's dimensions are similar)

Is this an "okay" method? If it is not, what might be a good way to handle this issue?

Note: Lowering all words in preprocessing is not an option for now.

Edit: The info about whether or not feature dimensions are truly linear would also be helpful.

Edit 2: Combining both answers from patapouf_ai and yazhi gave the best results. How are these combined? Weighted average improved the results but putting word frequencies through a scaled sigmoid function gave the best results, because using word frequencies in a linear manner gives them more importance than they bear.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

Just averaging them might not be good because that would assume that they they have the same weight, and that is probably not the case if the capitalized and uncapitilized version appear with very different frequencies in your training data.

An incremental improvement would be to average them proportionally to their frequency in the corpus. So say Earth appears 159 times and earth 1239 times do something like:

v(Earth & earth ) = 159/(159+1239) * v(Earth) + 1239/(159+1239) *v(earth).

The vectors are supposed to encode semantics linearly, so this should give you a resonable approximation.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ This is the solution we've chosen already. I might as well accep the answer. $\endgroup$
    – ozgur
    Commented Oct 4, 2016 at 7:53
0
$\begingroup$

The words "Earth" and "earth" may have the same meaning, but according to word2vec algorithm, it derives the semantic information from the position of the words.

Thus commonly, "Earth" will appear most often at the start of the sentence being a subject and "earth" will appear mostly in the object form at the end. So, the closest adjacent words may differ, but on overall both the sentences may contain the words such as "pollution, climate, water, countries".

In conclusion, I guess with a larger window size, it seems to preserve the same semantic information with a little changes where the "Earth" will have some subject information and "earth" will have object information. So, averaging wont affect much and seems to be a possible case. But with lower window size, there is a high probability that it could have different meanings.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Well, BoW size is 5. Which number do you consider an appropriate window size for averaging to work? $\endgroup$
    – ozgur
    Commented May 12, 2016 at 9:16
  • $\begingroup$ window size of 5 means, in total it considers 10 words and a common english sentence could be written in 10 words. So that sounds fine for me. $\endgroup$
    – chmodsss
    Commented May 12, 2016 at 9:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.