1
$\begingroup$

I'm trying to find a tool or workflow that would allow a team to analysis data in multiple steps, where intermediate results are cached. (I'll assume that the raw data is in a relational database and the end result shall be shown in a web UI.)

Multiple steps: This requirement can be describe in SQL as VIEWS:

CREATE VIEW b AS SELECT FROM a WHERE ... GROUP BY ... HAVING ...
CREATE VIEW c AS SELECT FROM b WHERE ... GROUP BY ... HAVING ...

Caching: Additionally to the process described above, we want to temporarily store the 'views' b and c. (Either until the underlying data changes or for a set amount of time. The latter is acceptable for the use case.)

Saving the view as a (temporary) tables, updated periodically, comes to mind, but we would have to build a workflow around this.

Team: This project requires some collaboration. A user should be able share queries with a team and have others build on top of it.

I want to avoid reinventing the wheel. Is there a technology stack that does this already?

$\endgroup$
1
  • 1
    $\begingroup$ Many RDBMs implement a cache mechanism. For instance, take a look to this information in the AWS website: aws.amazon.com/blogs/big-data/… $\endgroup$
    – Pablo Suau
    Commented Mar 10, 2017 at 11:55

0

Browse other questions tagged or ask your own question.