2
$\begingroup$

I have a pipeline which outputs model scores to s3. I need to partition the data by model_type and date. Which is the most efficient way to partition the data from the below:

  1. s3://bucket/data/model_type=foo/dt=YYYY-MM-DD/a.csv
  2. s3://bucket/data/dt=YYYY-MM-DD/model_type=foo/a.csv
Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

While doing partitioning of data we need to understand that too many partitions are not a good practice and partition only model type may lead too too few. So before you decide on best partitioning think about following:

  1. How will you access the data.. Which access patterns are more frequent. If you use model and then dates then probably go for first.
  2. If its date go for second one.
  3. Also instead of creating a partition for each day you may create biweekly partitions which will give you best of both worlds
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.