I have a big dataset that trying to train with a Doc2vec model. I am working on a 8 CPU, 32GB RAM, but as I can see on the monitoring tools, it only uses about 66-67% of the CPU. I am not sure if it is a matter of code or system preferences, but I have to start from something. So, here is my code

from gensim.models.doc2vec import Doc2Vec, TaggedDocument
from const import DOC2VEC_SIZE, STOP_WORDS
import dask.dataframe as ddf
import gensim
import multiprocessing

cores = multiprocessing.cpu_count()
assert gensim.models.doc2vec.FAST_VERSION > -1, "This will be painfully slow otherwise"

print('Number of cores used:', cores)

def clean(text):
    #do some text cleaning here
    return [...]

texts = ddf.read_csv('myCSV.csv', sep = ';', names = ['title', 'text'], skiprows = 1, error_bad_lines = False)

print('create tagged documents')

documents = texts.apply( lambda r: TaggedDocument(words=r['text'], tags=[r.title]), axis=1, meta=("text", "object") )

print('init doc2vec')

model =  Doc2Vec(dm=0, dbow_words=1, vector_size= DOC2VEC_SIZE, window=15, min_count=5, epochs=100, workers=cores)

print('build vocab')



model.train(documents, total_examples=len(documents), epochs=model.epochs)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.