Make Keras run on multi-machine multi-core cpu system

Question

I'm working on Seq2Seq model using LSTM from Keras (using Theano background) and I would like to parallelize the processes, because even few MBs of data need several hours for training.

It is clear that GPUs are far much better in parallelization than CPUs. At the moment, I only have CPUs to work with. I could access 16 CPUs(2 Threads per core X 4 cores per socket X 2 sockets)

From the doc of multi-core support in Theano, I managed to use all the four cores of a single socket. So, basically the CPU is at 400% usage with 4CPUs used and the remaining 12 CPUs remain unused. How do I make use of them too. Tensorflow could also be used instead of Theano background, if it works.

Answer 1

In order to set the number of threads used in Theano (and, therefore, the number of CPU cores), you'll need to set a few parameters in the environment:

import os
os.environ['MKL_NUM_THREADS'] = '16'
os.environ['GOTO_NUM_THREADS'] = '16'
os.environ['OMP_NUM_THREADS'] = '16'
os.eviron['openmp'] = 'True'

This should allow you to use all cores of all CPUs.

This can, of course, also be done in Tensorflow:

import tensorflow as tf
from keras.backend import tensorflow_backend as K

with tf.Session(config=tf.ConfigProto(
                    intra_op_parallelism_threads=16)) as sess:
    K.set_session(sess)
    <Your Keras code>