I don't have much experience in data engineering, so I'm here to ask for advice. I am working on a project which consists of building a dashboard for the IT department of a bank. the dashboard should present information from log data. Log data includes security vulnerabilities, issues reported by the help desk, and logs showing who is working on those issues. The data includes information such as description of issues, when they are reported, which device is affected.... Data is provided via an internal API (I don't believe it provides real-time data streaming). I want to create a data pipeline that extracts this data, transforms it, loads it into a database, and then creates a dashboard from it. Normally this pipeline should run once a day. so I think an ETL should work fine. I was thinking of using Python and Pandas to perform ETL since the data is not very large. The challenge is that alongside this ETL (which should be scheduled to run once a day), I want to achieve this functionality: If a critical issue is reported (server is down, high risk security vulnerability , ...) The IT department must be notified immediately (via the dashboard). How to implement such a pipeline. The data pipeline and dashboard must be deployed internally (no cloud services). Can you help me choose the right tools and give me some tips for designing this pipeline. THANKS.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.