Using several documents with word2vec

Question

I want to train word embeddings using word2vec. My corpus is split into several documents (it's a large set of patient notes). Should I just concatenate all documents into one before running word2vec on it, or is there a better way?

Answer 1

There are a number of implementations of Word2Vec, but most assume the basic unit to be 'sentences' - though they don't care what those sentences look like. If you are using something like gensim you will need each sentence in its own list, and each sentence will be a list of tokens. If you are using another package you may be reading from the disk in which case you likely need all documents concatenated with each newline representing a new sentence.

The fundamental consideration in deciding on the size of a 'sentence' (whether it's the entire note, all a patient's notes, or a single sentence) is what surrounding words should be used to train the model. The W2V model will consider all words within a certain distance of the target word when tuning its vector representation (regardless of if it's cbow or skipgram), but will not look beyond a 'sentence' boundary. So if your fundamental unit is a document not a sentence, associations will bleed across sentences (this may or may not be what you want).

As an aside, consider the nature of the documents first. Patient notes can be messy and full of automatically generated text, so it can be important to strip out or replace certain strings. Similarly, they often use whitespace and newlines rather than punctuation. You might want to consider treating existing whitespace as the end of a sentence and you might want to use a parser to replace dates, names, etc with default tokens to increase generalizability.

Answer 2

There is no need of concatenation. The iterator class can be changed accordingly so that the model could iterate over all the documents in a single folder.

import os

class WordTrainer(object):
   def __init__(self, dir_name):
      self.dir_name = dir_name
   def __iter__(self):
      for idx,file_name in enumerate(os.listdir(self.dir_name)):
        for idxx,line in enumerate(open(os.path.join(self.dir_name, file_name),'r'):
            words = [word.lower() for word in line.split()]
            yield words

 patient_details = WordTrainer('/path/for/records/folder')
 word_vector_model = gensim.models.Word2Vec(patient_details, size=100, window=8, min_count=5)

The iterator provides each data from the files line by line, rather than loading an entire document into the model. It is not a memory efficient approach when you have few documents that could fit into your memory.

Answer 3

I would suggest try doc2vec instead of word2vec. doc2vec not only gives you word embeddings as given by word2vec but it also gives you document embeddings. Using the document embeddings you can also do operations like "which patients have similar notes".

Needless to say, with doc2vec you can feed one document at a time.