Timeline for How to deal with version control of large amounts of (binary) data
Current License: CC BY-SA 3.0
34 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 17 at 12:23 | answer | added | mve | timeline score: 0 | |
| Sep 3, 2023 at 16:57 | answer | added | Srini Kadamati | timeline score: 0 | |
| Aug 15, 2019 at 6:14 | answer | added | hhsecond | timeline score: 0 | |
| Oct 13, 2017 at 9:22 | answer | added | Borg | timeline score: 0 | |
| May 13, 2017 at 23:07 | answer | added | Dmitry Petrov | timeline score: 9 | |
| Sep 29, 2015 at 21:23 | vote | accept | Johann | ||
| Aug 29, 2015 at 21:44 | answer | added | BChan | timeline score: 6 | |
| Aug 12, 2015 at 9:56 | answer | added | norok2 | timeline score: 13 | |
| Apr 10, 2015 at 8:37 | answer | added | seanv507 | timeline score: 2 | |
| Apr 8, 2015 at 19:21 | answer | added | Piotr Migdal | timeline score: 11 | |
| Feb 21, 2015 at 0:50 | comment | added | r_31415 | @Johann Are you sure you need to version control your dataset? If you are processing a big set of images, you usually want to version control the procedures you followed that led to a modified set of images. Therefore, you usually don't need to keep track the images themselves. If you want to restore this modified set in the future, obviously you only need to take the original dataset and apply the procedures you did according to some commit in your code. | |
| Feb 19, 2015 at 6:30 | answer | added | fritzo | timeline score: 9 | |
| S Feb 19, 2015 at 5:31 | history | edited | Aleksandr Blekh | CC BY-SA 3.0 |
Improved title, grammar, wording and formatting.
|
| S Feb 19, 2015 at 5:31 | history | suggested | Piotr Migdal | CC BY-SA 3.0 |
title and tags
|
| Feb 18, 2015 at 13:03 | review | Suggested edits | |||
| S Feb 19, 2015 at 5:31 | |||||
| Feb 18, 2015 at 12:37 | history | migrated | from academia.stackexchange.com (revisions) | ||
| Feb 18, 2015 at 12:33 | comment | added | Piotr Migdal | @Johann Data scientist have different backgrounds. Mine is in quantum mechanics, for example. The whole point here is that: 1. StackExchange discourages so-called boat questions and 2. its better to get best practices rather than how it is solved by people who had to solve it but had no idea. | |
| Feb 16, 2015 at 5:16 | comment | added | Johann | @PiotrMigdal 1) You are probably right, though I am still curious how scientist - without a background in data science - handle that situation, 2) No doubt, git cannot handle that kind of data. A diff per file would be enough, just to see if my data has changed (or I changed it inadvertently). The point is to have control/documentation over how my data changed through what action by whom. | |
| Feb 15, 2015 at 19:53 | comment | added | Piotr Migdal | @Johann 1. How you store or version control it is in domain of data science (regardless if you use it in academia, industry or for a hobby project). I really want to ensure the best answers and it is good to go where there are many experts in data. 2. My point was only that it might be not for git. (And, all in all, git is a filesystem). Do you want to diff per file, line, or what? | |
| Feb 15, 2015 at 19:35 | comment | added | Johann | @PiotrMigdal Going without version control kind of pains me for the reasons I've stated in the question (especially: "did my data change with out me noticing?" and then 2 days of diffing by hand). So I am looking for something smarter. | |
| Feb 15, 2015 at 19:33 | comment | added | Johann | @JukkaSuomela That actually sounds pretty good (at least much better than anything I've found so far). | |
| Feb 15, 2015 at 19:32 | comment | added | Johann | @DaveRose 1. Yes, I will hopefully add more experimental data and processed images, but not very often (maybe a few iterations); 2. Yes, that is part of my thesis; 3. Yes, others will hopefully use and change the data. | |
| Feb 15, 2015 at 19:22 | comment | added | Johann | @PiotrMigdal I don't know if this is off-topic, but this question does also not fit well with Databases.SE or DataScience.SE. I would like to know what other researchers/Institute do in practice to deal with this kind of problem - I've updated the question accordingly. | |
| Feb 13, 2015 at 14:02 | comment | added | Piotr Migdal | I'm voting to close this question as off-topic because it deals with data/databases rather than something specific to academia. The questions is great, and (IMHO) should be moved to DataScience.SE or (perhaps) Databases.SE. | |
| Feb 13, 2015 at 13:38 | answer | added | jakebeal | timeline score: 9 | |
| Feb 13, 2015 at 13:29 | comment | added | jakebeal | @JukkaSuomela I think it's a reasonable question when you've got very large datasets, if those datasets change frequently... in those cases, backup often is what's used as version control. | |
| Feb 13, 2015 at 13:01 | comment | added | Jukka Suomela | @PiotrMigdal: Are you seriously asking why people should use version control, instead of just having a bunch of files with backups?-) | |
| Feb 13, 2015 at 12:22 | comment | added | Piotr Migdal | @Johann Why not just files, without version control (but with backups)? | |
| Feb 13, 2015 at 11:48 | comment | added | ff524 | @JukkaSuomela I think you should post that as an answer, not a comment. | |
| Feb 13, 2015 at 11:30 | comment | added | Jukka Suomela | Just to protect against data corruption with image data, I periodically run a script that re-computes a checksum file with all files and their md5 checksums. The checksum file is then kept in git. Now I can immediately see with git diff if any of the checksums have changed. And I can also see which files have been removed & added. And if there are e.g. any signs of data corruption, then I can use the regular backups to restore old versions. Not perfect but better than nothing. | |
| Feb 13, 2015 at 11:01 | comment | added | ff524 | @scaaahu I don't think this is necessarily a software question; an acceptable answer could also describe a workflow or combination of tools and systems. (Anyways, being on topic somewhere else shouldn't play into the decision to close a question here.) | |
| Feb 13, 2015 at 10:52 | comment | added | scaaahu | I voted to "Leave Open" when reviewing Close Votes queue because I don't think this question is for specific situation. However, can anybody who is familiar with Software Recommendations SE tell us this question would fit into that site? | |
| Feb 13, 2015 at 10:40 | comment | added | Dave Rose | If you tell me three things I might have an answer! 1. Does your medium size data gets bigger? If so how periodically. 2. Do you use programming languages/frameworks to do pattern matching and data analysis? 3. Does anyone else use this data? If so do they change it? | |
| Feb 13, 2015 at 10:09 | history | asked | Johann | CC BY-SA 3.0 |