3
$\begingroup$

I would like to train my model faster and that's why would like to create a standalone remote machine with GPU support. The solution I am looking for is to have the data locally on my machine and to use the resources from the remote machine. Is this possible? I am checking my project for example locally on my laptop. When I use some remote Jupiter notebook, then it needs the data also to be on the remote machine. But after the model is trained I would like to commit it in git from my local machine. How to make this?

$\endgroup$
1
  • $\begingroup$ Have you tried uploading the files onto dropbox and using wget command in your python script to download it? $\endgroup$ Jul 10 '20 at 15:24
0
$\begingroup$

What sort of virtual machine are you using? If you are using Google colab for example you can connect it to your drive and add data from your drive.

$\endgroup$
5
  • $\begingroup$ I don't use any virtual machine. I have my laptop and have a personal computer. I want to connect them to use the power of my personal computer for GPU. $\endgroup$
    – Mutatos
    Jul 10 '20 at 13:27
  • $\begingroup$ What do you mean by a remote machine ? $\endgroup$
    – mirimo
    Jul 10 '20 at 13:28
  • $\begingroup$ my personal computer is a remote machine for my laptop. $\endgroup$
    – Mutatos
    Jul 10 '20 at 13:32
  • $\begingroup$ In that case, you'll need to transfer the data to your personal computer or use a common drive $\endgroup$
    – mirimo
    Jul 10 '20 at 13:36
  • $\begingroup$ No, this is not a clean solution. I need to try to mount somehow my local drive or sync the files automatically on changes. $\endgroup$
    – Mutatos
    Jul 10 '20 at 13:45
0
$\begingroup$

It is possible but not recommended. Storing data locally and performing compute on the remote server has the potential for increasing training time.

Two possible scenarios:

  1. All data can be loaded into memory on the remote server. That would require a single trip between local and remote.

  2. If data can not be loaded into memory at the same time, then there will have be repeated trips between local and remote. That transfer latency would greatly slow down training.

A better option would be to provision a remote server with enough disk space to store all of the data. Compress the data. Move all of the compressed data to the remote server. Uncompress the data. Then train the model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.