1
$\begingroup$

I am working concurrently with multiple very large datasets (10s-100s of GBs)). I signed up for Colab Pro+ thinking it is the best option. However, I face a significant bottleneck in getting data into Colab.

My options all seem very bad:

  1. Downloading from AWS (where the data is located) - very slow.
  2. Uploading data to Google Drive and mounting Drive using below code. This is also surprisingly very slow.
from google.colab import drive
drive.mount('/content/drive')
  1. Paying for a persistent server. Something like a persistent AWS SageMaker notebook. This is very expensive. With even a mediocre GPU, it comes out to $2k/mo.

What is the best solution here? I am ok with paying for a good option as long as its reasonable. Have any of the numerous MLOps startups developed a good solution to this?

Thanks!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Try Faculty Platform. I had a good experience with large datasets(10s of GB) with them.

Faculty Platform

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.