Migrating legacy data in Kafka to use a schema registry to support a streaming data pipeline

Question

(If this is not the correct Stack site for this question, please let me know - it seemed the best fit)

We currently have a 'legacy' system whose output are events coming in from a distributed application environment. We wish to set up a streaming data pipeline to extract value from the multiple TB of transaction data that we have. However, we have what (I feel) are several large roadblocks before we can do so. The current landscape looks as follows:

1. Numerous Java services produce unversioned events in JSON format into a single, partitioned topic.
2. These events are stored in Kafka with an (as of yet) infinite retention.
3. Other services consume the produced messages and apply some business rule to the given data.

The problems with this approach should be fairly obvious:

1. There is no versioning or schema evolution concept for the messages. Hence, consumers must be aware of the ever-changing schema and be adjusted manually whenever there is a breaking change (or even if new data needs to be considered).
2. Since all events of a service land in a single topic, consumers sometimes have to waste precious CPU cycles just filtering the message type that they actually wish to work with. While this makes some small use cases simpler, it is indicative of a larger domain modelling issue.

I am looking at possible choices that would allow us to move towards our goal of setting up a streaming pipeline using this data, but I'm having trouble finding resources that pertain to our use case.

So, my question is: How would I go about setting up a schema registry for the existing data? My first use case is to set up a small data lake using data from the available Kafka topics, as well as some relational data using CDC and provide our data scientists a starting point for explorative analysis.

One possible (but not very efficient way) that I can think of, is to add a transformer to each of our current topics and then split them into separate topic pertaining to the specific event type. This way we would be able at least use a schema moving forward that could be used for further ETL processes, but it would not help the source system. Is there a better way?

Answer 1

As I don't have all the information, here are some tips that I will modify thanks to your feedbacks.

Such complex problems needs a progressive and safe approach, combined with a long term vision:

• Good analysis of the legacy data: which are the most frequent structures, which are the most important ones, etc. This would require some classification using remarquable and common features (data size, structure, date, author,...).
• Then a block by block development, in order to make most cases meaningful and comparable to each other. This leads to cleaner data.
• Finally, a convergence solution to a optimal (if not, better) schema and source system. If some features are absent in specific data groups, it is important to identify them in order to detect the limits, potentially interpret them, and define the best strategies as possible.
• Not to mention that those strategies would direct the evolving schema. The more the strategy or strategy schedule is optimal in the long term, the less changes and difficult adaptations you would expect in the future.
• Most long term strategies are depending on the costs. Maybe a good investment in an expensive system could be better than keeping legacy systems in the long run. Very often, recurrent cost due to a bad system or schema can be way more expensive than good ones with time. We should also take into account the additional benefits a better system could have that would lower costs even more.