11
$\begingroup$

I'm totally new to word2vec so pls bear it with me. I have a set of text files each containing a set of tweets, between 1000-3000. I have chosen a common keyword ("kw1") and wants to find semantically relevant terms for "kw1" using word2vec. For example if the keyword is "apple" I would expect to see related terms such as "ipad" "os" "mac"... based on the input file. So this set of related terms for "kw1" would be different for each input file as word2vec would be trained on individual files (eg., 5 input files, run word2vec 5 times on each file).

My goal is to find sets of related terms for each input file given the common keyword ("kw1"), which would be used for some other purposes.

My questions/doubts are:

  • Does it make sense to use word2vec for a task like this? is it technically right to use considering the small size of an input file?

I have downloaded the code from code.google.com: https://code.google.com/p/word2vec/ and have just given it a dry run as follows:

 time ./word2vec -train $file -output vectors.bin -cbow 1 -size 200 -window 10 -negative 25 -hs 1 -sample 1e-3 -threads 12 -binary 1 -iter 50

./distance vectors.bin

  • From my results I saw I'm getting many noisy terms (stopwords) when I'm using the 'distance' tool to get related terms to "kw1". So I did remove stopwords and other noisy terms such as user mentions. But I haven't seen anywhere that word2vec requires cleaned input data...?

  • How do you choose right parameters? I see the results (from running the distance tool) varies greatly when i change parameters such as '-window', '-iter'. Which technique should I use to find the correct values for the parameters. (manual trial and error is not possible for me as I'll be scaling up the dataset).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
9
$\begingroup$

Word2Vec isn't a good choice for a dataset of such size. From researches I have seen, it will unleash its power if you feed at least couple of million of words, 3k tweets wouldn't be enough for a concise word similarity.

Improve this answer
$\endgroup$
7
  • $\begingroup$ do we need to remove stopwords as a data pre processing step? $\endgroup$
    – samsamara
    Commented Jan 11, 2016 at 12:38
  • 2
    $\begingroup$ No, in the word2vec approach you don't need to do that, since the algorithm itself relies on a broad context to find similarities in words, so stop words (most of which are prepositions, pronouns and such) are an important asses for algorithm. $\endgroup$
    – chewpakabra
    Commented Jan 11, 2016 at 17:09
  • $\begingroup$ but when i was using the distance tool to find most similar words to a given word, the version with stopwords removed gave me sensible words than the version without. can you guess what's this mean? $\endgroup$
    – samsamara
    Commented Apr 15, 2016 at 1:34
  • $\begingroup$ Probably, you are using too narrow context: if your model looks into, say, two words back and forward, you will be having up to 2 stopwords in context and that could give worse results. If you will broaden context (which will make model bigger and training time longer), with-stopwords model will give you better results, I assume. $\endgroup$
    – chewpakabra
    Commented Apr 15, 2016 at 8:33
  • $\begingroup$ thanks for the input, makes more sense now. Also since word2vec processes the input sentence by sentence, what would happen if i mix up the sentences in the input document? that should totally change output vectors right? Also again, given it's processing sent by sent, how does word2vec differ from doc2vec? thanks again. $\endgroup$
    – samsamara
    Commented Apr 15, 2016 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.