how to input the data set in to a word2vec by keras?

Question

I am new in using word2vec model, as a result, I do not know how I can prepare my dataset as an input for word2vec? I have searched a lot but the datasets in tutorials were in CSV format or just one txt file, but my dataset is in this structure: 2 folders one of these is blood cancer and the other one is breast cancer. each folder contains 1000 txt files which contain 40 sentences. I do not have any idea about I can create a vocabulary as an input for the word2vec model in keras with tensorflow backend? I use python 3.5 in ubuntu 17.10 Any guidance will be appreciated.

Answer 1

I have already searched for the solution and found this statements which are proper for this kind of datasets: at first, you should concatenate 2 folders in one folder to apply the code below:

import os
import gensim, logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s',      level=logging.INFO)

class MySentences(object):
    def __init__(self, dirname):
        self.dirname = './Dataset#2_BinaryClassClassification/lupus_alldataset/'

    def __iter__(self):
        for fname in os.listdir(self.dirname):
           for line in open(os.path.join(self.dirname, fname)):
               yield line.split()

sentences = MySentences('./DatasetBinaryClassClassification/alldataset/') # a memory-friendly iterator
model = gensim.models.Word2Vec(sentences,
    size=200,
    window=10,
    min_count=2,
    workers=5)

I found the 'class MySentences' in this site:enter link description here

I hope it is helpful.