I have a bunch of projects for my job that are largely unrelated except they use the same data, which is pretty big on disk in csv format. I want these to exist separately from each other and I usually try to use the cookie cutter data science model for project structure, and keep all my data in a data folder in the root of the project.

But because this dataset is big, I don't want to have ten copies of it in the root of these ten projects. I also don't want to put them in one big project sharing it because I feel like they don't belong together.

What's the best way to structure multiple different projects that all share the same large dataset?

  • 1
    $\begingroup$ Have you considered a database solution such as running postgres locally? $\endgroup$ Feb 14 '19 at 17:09
  • $\begingroup$ @StevenTheDataGuy I have, and I realize that technically probably the best solution. But I'm kind of invested in using csv because 1. I have a lot of code going back that relies on it and 2. At some point the code will need to be re-used by someone who will have the csv as a starting point and I want to make it as easy as possible for them to use my code $\endgroup$
    – jesseWUT
    Feb 15 '19 at 15:17

A database is the best option to share data across projects.

Another option is version control. Check the csv into version control. It could be git, GitHub, or data specific version control system.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.