I am reading up about lambda architecture.
It makes sense. we have queue based data ingestion. we have an in-memory store for data which is very new and we have HDFS for old data.
So we have our entire data set. in our system. very good.
but the architecture diagram shows that the merge layer is able to query both the batch layer and the speed layer in one shot.
How to do that?
Your batch layer is probably a map reduce job or a HIVE query. The speed layer query is probably a scala program which is execution on the spark.
Now how will you merge these?
Is there any guidance.