I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that:

  • It holds all your data in a central accessible location
  • It updates all depending data sets when data is added to or changed in a data set
  • It can run any transformation, as long as it runs in a Docker, and accepts a file as input and outputs a file as result
  • It versions all your data
  • It handles both modified data and newly added fractions of data
  • It can keep branches of your data sets when you are testing new transformation pipelines

It seems that Git can handle all of them. And maybe data is always larger in size than code then git-lfs was created for that purpose.

In contrast, Dolt provides a different direction that combines SQL and Git.

Do tools like Pachyderm apply nowadays in data science?

  • 1
    $\begingroup$ From a quick look I think that Pachyderm has a lot more capabilities than git-lfs: afaik git-lfs is just a link and a checksum of the file, it doesn't deal with the content, cannot do a diff, and the synchronization of the data with collaborators is not automatic $\endgroup$
    – Erwan
    Commented Jan 13, 2021 at 17:35
  • $\begingroup$ @Erwan Git is too slow(Linux commands like ls) when the file size is too large or there are zip files in the directory. I wonder how Pachyderm-like tools deal with this problem. $\endgroup$ Commented Jan 14, 2021 at 12:06
  • $\begingroup$ I have no experience with pachyderm and very little with git-lfs, but I could imagine that pachyderm solves a few of git-lfs problems. It's a good question imho, I hope somebody who knows these two tools will answer. $\endgroup$
    – Erwan
    Commented Jan 14, 2021 at 12:48

1 Answer 1


Git is designed for code.

Pachyderm is designed for machine learning assets: data, pipelines, and notebooks.

You can put machine learning assets into git. However, git will treat machine learning assets just as code primitives. One example is notebooks which are JSON. JSON in git quickly becomes difficult to manage. Pachyderm will manage them in notebook specific ways.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.