2
$\begingroup$

The problem: I've N classification models (independent), for each of these N models, I've different versions (eg: V0, V1, ..., Vfinal_production,Vexperimental). I'm looking for a way to store my datasets efficiently on the cloud (for redudency).

Note: We're not talking about BigData here.

Current Solution: Created a private GitHub repo. Made N directories and inside, pushed different dataset versions as different files.

Are there better solutions for this (because I feel VCS is an overkill for this problem)?

$\endgroup$
  • $\begingroup$ how much/why is data changing from model to model? Also, as a rule of thumb: you shouldn't get in the practice of committing data to git. $\endgroup$ – David Marx Jan 13 '18 at 8:34
  • $\begingroup$ Each independent models are independent programs serving a different purpose. If not VCS, what are the alternative strategies? $\endgroup$ – ngub05 Jan 13 '18 at 8:45
  • $\begingroup$ use a database to store your data with auditing tables to track when records get changed, and track your model training code from the data-pull forward in VCS. $\endgroup$ – David Marx Jan 13 '18 at 9:24
0
$\begingroup$

I used my existing directory structure with Git LFS and it works well.

Here are the advantages I got by using Git LFS other than the alternative solutions.

  • It saves space: Since Git LFS manages my dataset files (100MB to 1GB) with pointers and keeping only the minimum in the local memory.

  • It works seamlessly with a git enabled repository: I was planning to use some storage solution (like Amazon S3) specific for my datasets. But I would have to make extra efforts to manage multiple version and keep my dataset files in sync.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.