I have a requirement of ingesting huge amount of user specific data and then searching on it.

Trying to be low cost solution, I am leaning towards creating Kafka based ingestion pipelines and DynamoDb based indexing of the basic meta data in each ingested record.

One main aspect is to catalog the incoming data so the search tool can use it and search in the data.

How does one go by creating a data catalog. The incoming data is JSON or XML.

What does data cataloging entail, if it is to be done from scratch.

Appreciate any leads that provides me direction/s forward


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.