(If this is not the correct Stack site for this question, please let me know - it seemed the best fit)
We currently have a 'legacy' system whose output are events coming in from a distributed application environment. We wish to set up a streaming data pipeline to extract value from the multiple TB of transaction data that we have. However, we have what (I feel) are several large roadblocks before we can do so. The current landscape looks as follows:
- Numerous Java services produce unversioned events in JSON format into a single, partitioned topic.
- These events are stored in Kafka with an (as of yet) infinite retention.
- Other services consume the produced messages and apply some business rule to the given data.
The problems with this approach should be fairly obvious:
- There is no versioning or schema evolution concept for the messages. Hence, consumers must be aware of the ever-changing schema and be adjusted manually whenever there is a breaking change (or even if new data needs to be considered).
- Since all events of a service land in a single topic, consumers sometimes have to waste precious CPU cycles just filtering the message type that they actually wish to work with. While this makes some small use cases simpler, it is indicative of a larger domain modelling issue.
I am looking at possible choices that would allow us to move towards our goal of setting up a streaming pipeline using this data, but I'm having trouble finding resources that pertain to our use case.
So, my question is: How would I go about setting up a schema registry for the existing data? My first use case is to set up a small data lake using data from the available Kafka topics, as well as some relational data using CDC and provide our data scientists a starting point for explorative analysis.
One possible (but not very efficient way) that I can think of, is to add a transformer to each of our current topics and then split them into separate topic pertaining to the specific event type. This way we would be able at least use a schema moving forward that could be used for further ETL processes, but it would not help the source system. Is there a better way?