I don't have much experience in data engineering, so I'm here to ask for advice. I am working on a project which consists of building a dashboard for the IT department of a bank. the dashboard should present information from log data. Log data includes security vulnerabilities, issues reported by the help desk, and logs showing who is working on those issues. The data includes information such as description of issues, when they are reported, which device is affected.... Data is provided via an internal API (I don't believe it provides real-time data streaming). I want to create a data pipeline that extracts this data, transforms it, loads it into a database, and then creates a dashboard from it. Normally this pipeline should run once a day. so I think an ETL should work fine. I was thinking of using Python and Pandas to perform ETL since the data is not very large. The challenge is that alongside this ETL (which should be scheduled to run once a day), I want to achieve this functionality: If a critical issue is reported (server is down, high risk security vulnerability , ...) The IT department must be notified immediately (via the dashboard). How to implement such a pipeline. The data pipeline and dashboard must be deployed internally (no cloud services). Can you help me choose the right tools and give me some tips for designing this pipeline. THANKS.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
To achieve this deployment, consider containerizing your project and using a scheduler to orchestrate runs based on your timings. If you use Python, you can create your own image and then use a local Kubernetes instance for orchestration.
I would use a document database to store all the data, both imported and generated by the project and then serve a visualization tool from within the same microservice environment.
Consider using websockets to achieve real-time notifications, which can be triggered from a websocket cliet the ETL process itself or from higher-level code that runs after the ETL has provided the data.
