2
$\begingroup$

I work for a small data science consultancy firm and we are trying to standardize our project folder structure. We started from the cookiecutter structure which is a great base.

However one of the discussion point lies in the subfolders of the data folder, which is structured as:

  • Raw
  • Interim
  • Processed

Let's think about the following situations:

  1. The client gives you a manually extracted csv file -> This obviously goes into Raw
  2. You have acces to SQL databases and make a no-modification extract -> Still into Raw I guess?
  3. Because of very large databases, you create a semi-complex SQL query as base for a feature -> Is this Raw or Interim?

What are the best practices you apply? What would you recommend?

Ps: links to Github projects constructed following this kind of structure are very welcome

$\endgroup$
1
$\begingroup$

What are the best practices you apply? What would you recommend?

My practice is "kind of structured but quite different every time", and I don't recommend it ;)

I suspect that I'm not the only one but I don't have any stats.

Thanks for the link to Cookiecutter, this looks interesting! After reading a bit about it, it looks to me like a key criterion is this: "anyone should be able to reproduce the final products with only the code in src and the data in data/raw."

Based on that I would argue that:

  1. You have acces to SQL databases and make a no-modification extract -> Still into Raw I guess?

Yep, because this data cannot be obtained by processing some other part of the raw data.

  1. Because of very large databases, you create a semi-complex SQL query as base for a feature -> Is this Raw or Interim?

Assuming this query is run on the raw sql database, the resulting data goes to interim, because it can be re-generated by running the query again. The query itself should be stored in src/features I think.

| improve this answer | |
$\endgroup$
  • $\begingroup$ The quote is definitely a good criterion. I'm just hesitating on point 3. Intuitively it does make more sense in Interim, but the nice thing about putting into Raw is that everything can always be regenerated from Raw only (without DB-connection). However, like you mention, if the query is stored into src/features, it is still is 'regeneratable'. $\endgroup$ – Technologic Nov 4 '19 at 20:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.