I have two GPUs, NVIDIA GTX 1070 Ti. For using Keras with TensorFlow back-end, should I connect them with SLI or not? If not, then they will be treated separately, and one model will be trained on one card. These are the two options from what I understood so far. Thank you.
You need not connect GPUs via SLI. Keras and TensorFlow will take care of distributing batches across GPUs
Instead of SLI, if you use NV-link, Keras can use use GPU for merge as well. https://www.nvidia.com/en-us/data-center/nvlink/
From the documentation :
Specifically, this function implements single-machine multi-GPU data parallelism. It works in the following way:
Divide the model's input(s) into multiple sub-batches. Apply a model copy on each sub-batch. Every model copy is executed on a dedicated GPU. Concatenate the results (on CPU) into one big batch. E.g. if your batch_size is 64 and you use gpus=2, then we will divide the input into 2 sub-batches of 32 samples, process each sub-batch on one GPU, then return the full batch of 64 processed samples.
This induces quasi-linear speedup on up to 8 GPUs.
This function is only available with the TensorFlow backend for the time being.