I would like to train my model faster and that's why would like to create a standalone remote machine with GPU support. The solution I am looking for is to have the data locally on my machine and to use the resources from the remote machine. Is this possible? I am checking my project for example locally on my laptop. When I use some remote Jupiter notebook, then it needs the data also to be on the remote machine. But after the model is trained I would like to commit it in git from my local machine. How to make this?
It is possible but not recommended. Storing data locally and performing compute on the remote server has the potential for increasing training time.
Two possible scenarios:
All data can be loaded into memory on the remote server. That would require a single trip between local and remote.
If data can not be loaded into memory at the same time, then there will have be repeated trips between local and remote. That transfer latency would greatly slow down training.
A better option would be to provision a remote server with enough disk space to store all of the data. Compress the data. Move all of the compressed data to the remote server. Uncompress the data. Then train the model.