0
$\begingroup$

So I am implementing Word2Vec for the first time, and I have a set of training data that I would like to train a word2vec model on. Predictably, the problem is the dataset is rather large, and I have more limited computational power than I would like. This is a very common problem of course, but are there any ways to minimize the input text without too horribly affecting the performance?

For example, if I had the example sentences:

Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much. They were the last people you’d expect to be involved in anything strange or mysterious, because they just didn’t hold with such nonsense. Mr. Dursley was the director of a firm called runnings, which made drills. He was a big, beefy man with hardly any neck, although he did have a very large mustache.

I could do some thing like take out the stop words... but as I googled, that's a bad idea according to one link and a good idea according to another... so which is it? I also looked into text summarization but that seems harder than implementing word2vec. Another idea I naively had was to randomly take out X% of sentences from the text. But that would obviously be a performance hit in the model, and I'm not sure how big.

So, are there any general methods for trying to do this, and is stop word removal a way to do it?

$\endgroup$
2
$\begingroup$

I don't think there really is a right or wrong answer to the "removing stopwords" question. Some people will argue that throwing away information will reduce model performance, while others argue that it'll increase noise.

I personally follow a simple rule of thumb. If my model depends on sentence structure, then I keep stopwords. If i'm modeling topics and more interested in important phrases, then I remove them. This seems to work well.

If you're looking for ways to reduce you matrix, then yes removing stopwords is a perfectly acceptable idea. Another thing you can do is apply common feature reduction techniques like LSA or chi2 to find the most important words and reduce your input space to the most meaningful words. However, doing this may dramatically effect the performance or your word2vec model. But if it is your only choice, then why not give it a go.

$\endgroup$
  • $\begingroup$ Good info, thanks! When you say LSA, you are referring to this correct? $\endgroup$ – ocean800 Jan 23 '18 at 0:43
  • $\begingroup$ Yep! That is what i'm referring too. $\endgroup$ – Tophat Jan 23 '18 at 11:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.