0
$\begingroup$

I have two GPUs, NVIDIA GTX 1070 Ti. For using Keras with TensorFlow back-end, should I connect them with SLI or not? If not, then they will be treated separately, and one model will be trained on one card. These are the two options from what I understood so far. Thank you.

$\endgroup$
0
$\begingroup$

You need not connect GPUs via SLI. Keras and TensorFlow will take care of distributing batches across GPUs

https://keras.io/utils/#multi_gpu_model

Instead of SLI, if you use NV-link, Keras can use use GPU for merge as well. https://www.nvidia.com/en-us/data-center/nvlink/

From the documentation :

Specifically, this function implements single-machine multi-GPU data parallelism. It works in the following way:

Divide the model's input(s) into multiple sub-batches. Apply a model copy on each sub-batch. Every model copy is executed on a dedicated GPU. Concatenate the results (on CPU) into one big batch. E.g. if your batch_size is 64 and you use gpus=2, then we will divide the input into 2 sub-batches of 32 samples, process each sub-batch on one GPU, then return the full batch of 64 processed samples.

This induces quasi-linear speedup on up to 8 GPUs.

This function is only available with the TensorFlow backend for the time being.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.