I am working on a small project and I would like to use the word2vec technique as a text representation method. I need to classify patents but I have only a few of them labelled and to increase the performance of my ML model, I would like to increase the corpus/vocabulary of my model by using a large amount of patents. The question is, once I have train my word embedding feature, how to use this larger corpus with my training data - my labelled data?

My data set is composed by 2000 patents which are labelled.

The patents used to train my word embedding corpus are 3 millions (some of my 2000 labelled patents are already included in this larger corpus) which I trained using Gensim.

Do you have any suggestions on how to do it?

Thank you very much in advance.


  • $\begingroup$ Hi Rob, welcome to the community :) Word2Vec is trained without any labelling necessary, so you can train it on the biggest dataset you can get. For your classification task, you can use the word embeddings from Word2Vec to train a model on top which will learn to classify using your labelled dataset. $\endgroup$ – Adam Oudad Jul 15 at 19:27
  • $\begingroup$ Thank you very much Adam. I have already trained the Word2Vec but I am not able yet to use it as corpus for my labelled dataset using Python. $\endgroup$ – Rob C Jul 16 at 14:46

Use large amount of un-label data to finetune the BERT based model. You can train BERT in unsupervised manner. Then, use that bert to get embeddings of your input text of label data and train a classifier.

| improve this answer | |
  • $\begingroup$ Hi Uday, thank you for your reply. Do you have a link that you could suggest me to use Bert running on Python? Cheers $\endgroup$ – Rob C Jul 16 at 14:48
  • $\begingroup$ Check transformers library from huggingface @RobC $\endgroup$ – Uday Jul 16 at 15:28
  • $\begingroup$ Thank you I will have a look at it. $\endgroup$ – Rob C Jul 17 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.