Do I need a datalake in my use case?

Question

My web application stores usage data, for example:

• tickets opened an closed
• tasks executed
• user scores

etc. I need to show dashboards and reports for usage and performance trends, like:

• How many tickets where opened/closed in a period?
• what is the average tasks execution time?
• who is the most active user?
• which user gets the best score?

etc. the set of questions is open, we can invent more.

Does this scenario fall under a datawarehouse classic approach? Data are coming from a relational database with a specific data model.

Somebody is asking me to create a datalake, just giving him all my raw data. I don't understand how does a datalake fit in this picture. With no model and relationship my data are usueless.

Also, I don't have a stream of events: I have tables in a relational database. For example a row in a "task" table will have an "open time" column and a "close time" column.

I'm confused about the need or advantages of creating a datalake vs a traditional datawarehouse approach

Answer 1

The point of a "Data Lake" is to store unstructured data together, along with the related meta-data, all in one place. The data is usually stored in the format in which it was recorded i.e. the raw format. Data is then only extracted and processed for usage as required.

There is a nice article that compares data warehouses and data lakes in an unbiased and practical manner.

---

From your description of your data and the problems you want to solve, I don't personally think a data lake would be required.

Perhaps your colleague is planning to create a data lake for larger purposes, e.g. to house the data for many projects such as yours, which might also want to use your data? There are the issues of security, data ownership and data stewardship to think about in this case. Questions to answer would be:

• who actually owns the data?
• who is allowed to use what data?
• who is allowed to view what data?
• who is responsible for ensuring data is available at all times?

Answer 2

In your scenario, a data lake is not required. It is used in Big Data applications where you have a lot of unstructured data that needs to be stored cheaply in its raw format. Since your application already uses relational databases, which is clearly structured data, you can use a data warehouse instead and stick to the format. Since you are using the data for visualization and reports, data warehouse is the better way to go since most BI tools, like Power BI, are a lot easier to work with. If you use a data lake, you need an ETL pipeline in between.