0
$\begingroup$

If I'm doing simple aggregation dashboards, what's the minimum amount of data which justifies building a full blown data processing pipeline? Is it reasonable to build a complicated pipeline with Mysql -> Hadoop -> Redshift and then analytics and visualizations on top of redshift if my database only has about 10million transactions per month. Would it be better to just run something like ReDash visualizations on top of my unaggregated data in MySql?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

You should only complicate it as much as necessary to meet your requirements.

  1. What kind of aggregations are you doing? Most DBs can handle basic counts pretty easily. 10 million in a month sounds like something a large MySQL instance could handle.
  2. How quickly does everything have to update? If the users only expect the summarized data to be updated monthly, you have a lot of time. If you have to update it more often, it might make sense to have MySQL collecting it but Redshift presenting it. Redshift's analytical focus and columnar storage means you may not need pre-aggregated data until your volume gets much higher than what would require aggregation in MySQL. I was using a 4 node Redshift cluster with the smallest instance types and it could do a count across >300 million rows in 17 seconds.

Can you not build aggregate tables in MySQL? What is making you want to put Hadoop in between MySQL and Redshift?

Improve this answer
$\endgroup$
2
  • $\begingroup$ I want to use Hadoop because: - I can do cleanup and some normalization of data there (e.g. mapping geo coordinates to cities) - I want the analytics to be updated daily and I want the visualizations to load fast, on unaggreated data they dashboard would become very slow - This pipeline would be future proof, It will allow us to do more complicated stuff in the future - Sounds much cooler than running plain queires on mysql tbh :P $\endgroup$
    – Furqan Shakoor
    Commented Aug 2, 2017 at 22:56
  • $\begingroup$ Daily updates in a regular RDBMS is not bad unless you are always recalculating all your history. $\endgroup$
    – CalZ
    Commented Aug 3, 2017 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.