I have a requirement of ingesting huge amount of user specific data and then searching on it.
Trying to be low cost solution, I am leaning towards creating Kafka based ingestion pipelines and DynamoDb based indexing of the basic meta data in each ingested record.
One main aspect is to catalog the incoming data so the search tool can use it and search in the data.
How does one go by creating a data catalog. The incoming data is JSON or XML.
What does data cataloging entail, if it is to be done from scratch.
Appreciate any leads that provides me direction/s forward