2
$\begingroup$

I would like to train my model faster and that's why would like to create a standalone remote machine with GPU support. The solution I am looking for is to have the data locally on my machine and to use the resources from the remote machine. Is this possible? I am checking my project for example locally on my laptop. When I use some remote Jupiter notebook, then it needs the data also to be on the remote machine. But after the model is trained I would like to commit it in git from my local machine. How to make this?

Improve this question
$\endgroup$
7
  • $\begingroup$ What sort of virtual machine are you using? If you are using Google colab for example you can connect it to your drive and add data from your drive. $\endgroup$
    – mirimo
    Jul 10, 2020 at 13:03
  • $\begingroup$ I don't use any virtual machine. I have my laptop and have a personal computer. I want to connect them to use the power of my personal computer for GPU. $\endgroup$
    – Mutatos
    Jul 10, 2020 at 13:27
  • $\begingroup$ What do you mean by a remote machine ? $\endgroup$
    – mirimo
    Jul 10, 2020 at 13:28
  • $\begingroup$ my personal computer is a remote machine for my laptop. $\endgroup$
    – Mutatos
    Jul 10, 2020 at 13:32
  • $\begingroup$ In that case, you'll need to transfer the data to your personal computer or use a common drive $\endgroup$
    – mirimo
    Jul 10, 2020 at 13:36

1 Answer 1

Reset to default
1
$\begingroup$

It is possible but not recommended. Storing data locally and performing compute on the remote server has the potential for increasing training time.

Two possible scenarios:

  1. All data can be loaded into memory on the remote server. That would require a single trip between local and remote.

  2. If data can not be loaded into memory at the same time, then there will have be repeated trips between local and remote. That transfer latency would greatly slow down training.

A better option would be to provision a remote server with enough disk space to store all of the data. Compress the data. Move all of the compressed data to the remote server. Uncompress the data. Then train the model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.