7
$\begingroup$

I am reading up about lambda architecture.

It makes sense. we have queue based data ingestion. we have an in-memory store for data which is very new and we have HDFS for old data.

So we have our entire data set. in our system. very good.

but the architecture diagram shows that the merge layer is able to query both the batch layer and the speed layer in one shot.

How to do that?

Your batch layer is probably a map reduce job or a HIVE query. The speed layer query is probably a scala program which is execution on the spark.

Now how will you merge these?

Is there any guidance.

Improve this question
$\endgroup$
2
  • $\begingroup$ You are likely querying the last known output of batch, not running a batch process. $\endgroup$
    – Sean Owen
    Jan 3, 2015 at 18:52
  • $\begingroup$ OK. so how do I merge the lats known output of batch with the streaming data stored inside spark discrete RDD? $\endgroup$
    – Knows Not Much
    Jan 3, 2015 at 19:02

2 Answers 2

Reset to default
3
$\begingroup$

What you are asking about is, in my view, the main problem of implementing a lambda architecture. Here are some suggestions on how to solve it.

The combination of Spark and Spark Streaming largely supersedes the original lambda architecture (which usually involved Hadoop and Storm). Read here an example of how to use a SparkContext and a separate StreamingContext to produce different RDDs, one for batch processed results and another for real-time results.

Once you have replicated that in your system, you still have to think about how to query both kind of RDDs. The trivial case would be to just union both of them:

scala> rdd1.union(rdd2).collect

Or maybe you can create a new DStream, similar to stateStream in the linked example, where some keys are kept for real-time results, and others for batch results.

Improve this answer
$\endgroup$
2
  • $\begingroup$ this means Lambda architecture a little bit of airy fairy thing. easy to talk on slides and looks pretty but then in reality it not so easy to implement. $\endgroup$
    – Knows Not Much
    Jan 9, 2015 at 15:11
  • $\begingroup$ or a better analogy is the mice deciding to "bell the cat". great architecture... but who is going to do it? $\endgroup$
    – Knows Not Much
    Jan 9, 2015 at 15:13
3
$\begingroup$

From what I understand of the objectives of the lambda architecture your point:

Your batch layer is probably a map reduce job or a HIVE query.

Is not what was intended. The batch layer is not meant to be directly queried against, but rather feeds a serving layer, possibly a simple key-value store, for low latency queries.

lambda architecture diagram

Check out http://lambda-architecture.net/ for a more full explanation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.