I am new in using word2vec model, as a result, I do not know how I can prepare my dataset as an input for word2vec? I have searched a lot but the datasets in tutorials were in CSV format or just one txt file, but my dataset is in this structure: 2 folders one of these is blood cancer and the other one is breast cancer. each folder contains 1000 txt files which contain 40 sentences. I do not have any idea about I can create a vocabulary as an input for the word2vec model in keras with tensorflow backend? I use python 3.5 in ubuntu 17.10 Any guidance will be appreciated.


I have already searched for the solution and found this statements which are proper for this kind of datasets: at first, you should concatenate 2 folders in one folder to apply the code below:

import os
import gensim, logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s',      level=logging.INFO)

class MySentences(object):
    def __init__(self, dirname):
        self.dirname = './Dataset#2_BinaryClassClassification/lupus_alldataset/'

    def __iter__(self):
        for fname in os.listdir(self.dirname):
           for line in open(os.path.join(self.dirname, fname)):
               yield line.split()

sentences = MySentences('./DatasetBinaryClassClassification/alldataset/') # a memory-friendly iterator
model = gensim.models.Word2Vec(sentences,

I found the 'class MySentences' in this site:enter link description here

I hope it is helpful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.