Skip to main content
Wording improved
Source Link
Esmailian
  • 9.4k
  • 2
  • 32
  • 48

Can I use doc2vec for classification big documents (500-2000 words, 20000 total documents, classification for three classes)?

Yes. This amount of documents is considered small to medium. It is worth noting that the original Doc2Vec paper experimented with 75K documents.

Is it a problem that the documents are large enough and contain many common words?

No, it is OK. Consider the fact that both common and distinctive words are increasing bywith the size of documents, and distinctive words are what that matters. That is, larger texts are easier to distinguish.

Can I train my data together with Wikipedia articles (using unique tag for every article) for a more accurate calculation word-embeddings, or it can't give positive effect?

It depends and is worth the try. If Wikipedia documents are close in format and content to your documents, they could definitely help. In the Doc2Vec paper, authors used 25K labeled documents (IMDb reviews with positive and negative labels) and 50K unlabeled ones for training.

Also, this is a nice tutorial on using doc2vec. It reports 1-2 hours for 100K documents.

Can I use doc2vec for classification big documents (500-2000 words, 20000 total documents, classification for three classes)?

Yes. This amount of documents is considered small to medium. It is worth noting that the original Doc2Vec paper experimented with 75K documents.

Is it a problem that the documents are large enough and contain many common words?

No, it is OK. Consider the fact that both common and distinctive words are increasing by the size of documents, and distinctive words are what that matters. That is, larger texts are easier to distinguish.

Can I train my data together with Wikipedia articles (using unique tag for every article) for a more accurate calculation word-embeddings, or it can't give positive effect?

It depends and is worth the try. If Wikipedia documents are close in format and content to your documents, they could definitely help. In the Doc2Vec paper, authors used 25K labeled documents (IMDb reviews with positive and negative labels) and 50K unlabeled ones for training.

Also, this is a nice tutorial on using doc2vec. It reports 1-2 hours for 100K documents.

Can I use doc2vec for classification big documents (500-2000 words, 20000 total documents, classification for three classes)?

Yes. This amount of documents is considered small to medium. It is worth noting that the original Doc2Vec paper experimented with 75K documents.

Is it a problem that the documents are large enough and contain many common words?

No, it is OK. Consider the fact that both common and distinctive words are increasing with the size of documents, and distinctive words are what that matters. That is, larger texts are easier to distinguish.

Can I train my data together with Wikipedia articles (using unique tag for every article) for a more accurate calculation word-embeddings, or it can't give positive effect?

It depends and is worth the try. If Wikipedia documents are close in format and content to your documents, they could definitely help. In the Doc2Vec paper, authors used 25K labeled documents (IMDb reviews with positive and negative labels) and 50K unlabeled ones for training.

Also, this is a nice tutorial on using doc2vec. It reports 1-2 hours for 100K documents.

Source Link
Esmailian
  • 9.4k
  • 2
  • 32
  • 48

Can I use doc2vec for classification big documents (500-2000 words, 20000 total documents, classification for three classes)?

Yes. This amount of documents is considered small to medium. It is worth noting that the original Doc2Vec paper experimented with 75K documents.

Is it a problem that the documents are large enough and contain many common words?

No, it is OK. Consider the fact that both common and distinctive words are increasing by the size of documents, and distinctive words are what that matters. That is, larger texts are easier to distinguish.

Can I train my data together with Wikipedia articles (using unique tag for every article) for a more accurate calculation word-embeddings, or it can't give positive effect?

It depends and is worth the try. If Wikipedia documents are close in format and content to your documents, they could definitely help. In the Doc2Vec paper, authors used 25K labeled documents (IMDb reviews with positive and negative labels) and 50K unlabeled ones for training.

Also, this is a nice tutorial on using doc2vec. It reports 1-2 hours for 100K documents.