I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that:

  • It holds all your data in a central accessible location
  • It updates all depending data sets when data is added to or changed in a data set
  • It can run any transformation, as long as it runs in a Docker, and accepts a file as input and outputs a file as result
  • It versions all your data
  • It handles both modified data and newly added fractions of data
  • It can keep branches of your data sets when you are testing new transformation pipelines

It seems that Git can handle all of them. And maybe data is always larger in size than code then git-lfs was created for that purpose.

Do tools like Pachyderm apply nowadays in data science?

  • $\begingroup$ From a quick look I think that Pachyderm has a lot more capabilities than git-lfs: afaik git-lfs is just a link and a checksum of the file, it doesn't deal with the content, cannot do a diff, and the synchronization of the data with collaborators is not automatic $\endgroup$ – Erwan Jan 13 at 17:35
  • $\begingroup$ @Erwan Git is too slow(Linux commands like ls) when the file size is too large or there are zip files in the directory. I wonder how Pachyderm-like tools deal with this problem. $\endgroup$ – Lerner Zhang Jan 14 at 12:06
  • $\begingroup$ I have no experience with pachyderm and very little with git-lfs, but I could imagine that pachyderm solves a few of git-lfs problems. It's a good question imho, I hope somebody who knows these two tools will answer. $\endgroup$ – Erwan Jan 14 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.