The problem: We have several people that do a lot of ML work in our lab and have more people asking to use our equipment. We have several machines that have Ubuntu on them that people share. This causes a problem for resource contention. PersonA had all of their code/libraries on MachineA or a person is waiting on MachineB which has 4GPUs instead of 1GPU.
Looking for: What I’m looking for is a way of creating a cluster with these machines that will allow people to utilize and share the resources. This would allow people to scale out and make better use of the resources that we already have (e.g. waiting for a specific machine that is already used). This could be used by interactively running code in their IDE or submitting jobs to the cluster like some type of scheduler (slurm, pbs). At this point the end user could look at this as a "pool" of resources. (e.g. we have 100GPUs, personA requests 4GPUs, we have 96GPUs available for someone else to request). Would also like to avoid the user from having to anything about the cluster. For example UserA simply requests 4GPUs, they don't know and don't care which machines it gets launched on.
They use a mix of Tenserflow and mxnet with keras and some theano.
Tensorflow/mxnet seem to have clustering option but don’t think this would handle the scheduling problem? or would it?
If you have successfully deployed some type of machine learning cluster with GPU nodes, I would really be interested in seeing your architect, tools, and software you used.
If this is out of scope for this forum please feel free to move this to another channel.
