I want to train word embeddings using word2vec. My corpus is split into several documents (it's a large set of patient notes). Should I just concatenate all documents into one before running word2vec on it, or is there a better way?

  • $\begingroup$ Do these documents relevant to the same domain? Or each document explaining different things, that are not related wih other documents. $\endgroup$
    – Nomiluks
    Apr 7 '16 at 21:38
  • $\begingroup$ @Nomi it's a large set of patient notes: same domain, but different patients. $\endgroup$ Apr 7 '16 at 21:44
  • $\begingroup$ If domain is same then, you can train word2vec over all the sentences in the patient files. Either you mix all files sentences or or train model file by file in single run. $\endgroup$
    – Nomiluks
    Apr 8 '16 at 5:49
  • $\begingroup$ Also checkout TF-IDF $\endgroup$
    – Aditya
    Feb 28 '18 at 2:32
  • $\begingroup$ Thank you @yazhi for sharing iterator class code with us but my data set is something different as I have 2 folders which one of these is blood cancer and the other one is breast cancer. Each folder contains 1000 txt files which each file contains 40 sentences. and I think the above code "WordTrainer", is suitable for one folder, would you mind plz help me with creating a proper structure data set as input for word2vec model in keras with tensorflow backend. I use python 3.5 on ubuntu 17.10 I am new in using word2vec model in keras, any guidance will be appreciated. $\endgroup$
    – Maryam
    Jun 19 '18 at 22:48

There are a number of implementations of Word2Vec, but most assume the basic unit to be 'sentences' - though they don't care what those sentences look like. If you are using something like gensim you will need each sentence in its own list, and each sentence will be a list of tokens. If you are using another package you may be reading from the disk in which case you likely need all documents concatenated with each newline representing a new sentence.

The fundamental consideration in deciding on the size of a 'sentence' (whether it's the entire note, all a patient's notes, or a single sentence) is what surrounding words should be used to train the model. The W2V model will consider all words within a certain distance of the target word when tuning its vector representation (regardless of if it's cbow or skipgram), but will not look beyond a 'sentence' boundary. So if your fundamental unit is a document not a sentence, associations will bleed across sentences (this may or may not be what you want).

As an aside, consider the nature of the documents first. Patient notes can be messy and full of automatically generated text, so it can be important to strip out or replace certain strings. Similarly, they often use whitespace and newlines rather than punctuation. You might want to consider treating existing whitespace as the end of a sentence and you might want to use a parser to replace dates, names, etc with default tokens to increase generalizability.


There is no need of concatenation. The iterator class can be changed accordingly so that the model could iterate over all the documents in a single folder.

import os

class WordTrainer(object):
   def __init__(self, dir_name):
      self.dir_name = dir_name
   def __iter__(self):
      for idx,file_name in enumerate(os.listdir(self.dir_name)):
        for idxx,line in enumerate(open(os.path.join(self.dir_name, file_name),'r'):
            words = [word.lower() for word in line.split()]
            yield words

 patient_details = WordTrainer('/path/for/records/folder')
 word_vector_model = gensim.models.Word2Vec(patient_details, size=100, window=8, min_count=5)

The iterator provides each data from the files line by line, rather than loading an entire document into the model. It is not a memory efficient approach when you have few documents that could fit into your memory.

  • $\begingroup$ It is less efficient if all documents can fit in memory (I assume it's the case nowadays with a lot of RAM available). The reason is that the model makes several passes over the data, so here it will need to re-read it each time. $\endgroup$ Nov 12 '16 at 6:24
  • $\begingroup$ The higher the amount of data, the better would be the quality of the word vectors generated. In that case, I doubt that all those data could be loaded into the memory at once. And yes, im wrong, it is not memory efficient one, but a lot of time my system froze due to loading of all the data at once and sometimes multitasking is impossible. $\endgroup$
    – chmodsss
    Nov 12 '16 at 11:59

I would suggest try doc2vec instead of word2vec. doc2vec not only gives you word embeddings as given by word2vec but it also gives you document embeddings. Using the document embeddings you can also do operations like "which patients have similar notes".

Needless to say, with doc2vec you can feed one document at a time.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.