2
$\begingroup$

I am using low level Tensorflow API's for my model training. When I say low level it means I'm defining the tf.Session() object of the graph and evaluate graph with in this session.

I would like to distribute the model training using tf.distribute.MirroredStrategy().

I am able to use mirroredstrategy() on tensorflow sequential API's using the example shared by tensorflow in their document.

But I am facing difficulty in executing tf low level code using mirror strategy.

I tried to use distribute.MirrorStrategy() and below are the results of resource utilization:

[0] GeForce RTX 2080 Ti | 48'C, 40 % | 10771 / 11019 MB | vipin(10763M) gdm(4M)
[1] GeForce RTX 2080 Ti | 37'C, 0 % | 10376 / 11014 MB | vipin(10327M) gdm(36M) gdm(8M)

Even though model used the memory of both the GPU's, but still GPU1 utilization is 0.

I am not sure about the issue. Even not sure if tensorflow support this.

Please clear my doubts and if possible share the sample code as well.

Improve this question
$\endgroup$
1
  • $\begingroup$ Can you share some code that leads up to the above? In this way w emay be able to help more. $\endgroup$
    – hH1sG0n3
    Commented Jun 17, 2021 at 11:05

1 Answer 1

Reset to default
1
$\begingroup$

1- Use distribute.MirrorStrategy() at the beginning of your code (just after library declaration) to activate correctly all the related functions.

2- Set env variable CUDA_VISIBLE_DEVICES=1 (https://stackoverflow.com/questions/37893755/tensorflow-set-cuda-visible-devices-within-jupyter)

3- Ensure that mirrored variables are set correctly thanks to:

with strategy.scope():
     x = tf.Variable(..)
Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for your response. Rather I want to utlize my both the available GPU's using Tensorflow Mirror Strategy API's. As I mentioned, I'm suing Low Level Tensorflow API's and defining the session() object and then using sess.run(), executing the model. Even I have defined my whole graph under startegy.scope(). But still no luck. $\endgroup$
    – vipin bansal
    Commented Jun 23, 2021 at 3:32
  • $\begingroup$ Did you force strategy to GPU 1 only to see if it can be activated? It could be a hardware issue. You can test if the GPU 1 is KO thanks to gpu-burn: github.com/wilicc/gpu-burn $\endgroup$
    – Nicolas Martin
    Commented Jun 23, 2021 at 5:52
  • $\begingroup$ Thinks are working fine, if Im using Tensorflow Sequential API. Not sure if Mirror Strategy works for low level API's? $\endgroup$
    – vipin bansal
    Commented Jun 23, 2021 at 7:06
  • $\begingroup$ What is the batch size? Did you use experimental_distribute_datasets_from_function inside with strategy.scope() to split manually your data? Without reading the code, it's more difficult to see what is going wrong. Here is the most complete guide I've found about mirrored strategy: youtube.com/watch?v=jKV53r9-H14 $\endgroup$
    – Nicolas Martin
    Commented Jun 23, 2021 at 7:42
  • $\begingroup$ Thanks for the link. Meanwhile if you have 2 gpus machine, you can try some basic code . For me its difficult to share the code due to some restrictions. $\endgroup$
    – vipin bansal
    Commented Jun 24, 2021 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.