Reducing text input size into word2vec without affecting performance too badly?

Question

So I am implementing Word2Vec for the first time, and I have a set of training data that I would like to train a word2vec model on. Predictably, the problem is the dataset is rather large, and I have more limited computational power than I would like. This is a very common problem of course, but are there any ways to minimize the input text without too horribly affecting the performance?

For example, if I had the example sentences:

Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much. They were the last people you’d expect to be involved in anything strange or mysterious, because they just didn’t hold with such nonsense. Mr. Dursley was the director of a firm called runnings, which made drills. He was a big, beefy man with hardly any neck, although he did have a very large mustache.

I could do some thing like take out the stop words... but as I googled, that's a bad idea according to one link and a good idea according to another... so which is it? I also looked into text summarization but that seems harder than implementing word2vec. Another idea I naively had was to randomly take out X% of sentences from the text. But that would obviously be a performance hit in the model, and I'm not sure how big.

So, are there any general methods for trying to do this, and is stop word removal a way to do it?

Answer 1

I don't think there really is a right or wrong answer to the "removing stopwords" question. Some people will argue that throwing away information will reduce model performance, while others argue that it'll increase noise.

I personally follow a simple rule of thumb. If my model depends on sentence structure, then I keep stopwords. If i'm modeling topics and more interested in important phrases, then I remove them. This seems to work well.

If you're looking for ways to reduce you matrix, then yes removing stopwords is a perfectly acceptable idea. Another thing you can do is apply common feature reduction techniques like LSA or chi2 to find the most important words and reduce your input space to the most meaningful words. However, doing this may dramatically effect the performance or your word2vec model. But if it is your only choice, then why not give it a go.