I am somewhat confident this is the correct approach but I need a little sanity check before setting out on this.

I want to record a numerical value for a category over time. There will be many categories and may appear as follows:

1  cat_1
4  cat_2
10 cat_3
4  cat_4
10 cat_5
8  cat_6
99 cat_7
4  cat_8

Say, the count (on the left) on the left will be different each time I poll for the data.

I will want to run analysis on this later on.

My plan is to store the data each time I poll for it in a JSON encoded object in S3 (along with the timestamp). This means it will be in raw form.

Later on I will want to query that information to see the values per category over time.

Will this be sufficient and easy to analyse later?


It sounds like you're planning to store all the input data. That has to be sufficient if your problem can be solved. There is nothing else...

Whether it's "easy", that's a relative term and it depends on many things. It depends on the kind of analysis that you want to do and the amount of data, among other things. If you have less then a few MB of data and you just want to calculate basic statistics like the mean or the sum, then Excel is much easier than S3. But S3 is very scalable, and it can easily be read from systems that allow to train complex models, so if you have a lot of data and that's what you want to do, then it could be a good choice.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.