2
$\begingroup$

I have a 500MB model which I am commiting to Git. That is a really bad practice since for newer model versions the repository will be huge. As well, It will slow down all builds for deployments.

I thought of using another repository that contains all the models and then fetch them in running time.

Does anybody know a clean approach or alternative?

$\endgroup$
1
$\begingroup$

In most cases, you would use a file-storage solution such as Amazon S3 or Google Cloud and many others, which provide designated solutions for large object storage and retrieval.

You would then ideally want to update your code to retrieve the model directly from the file storage. Whether this download needs be done on every run or only once (storing the model locally for future runs), should be decided based on your specific needs.

$\endgroup$
  • $\begingroup$ Sounds great. What if I always need to fetch the latest version of the model without changing the code? Maybe adding it in the storage as model_latest.pt and rename the previous version? $\endgroup$ – room13 Feb 5 '20 at 9:51
  • $\begingroup$ Yes that is absolutely viable. These services also allow you to list directories so you can get the latest uploaded file by date. $\endgroup$ – Shushan Feb 5 '20 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.