4
$\begingroup$

Looking at the first paper on RDDs/Apache Spark, I found a statement saying that "RDDs degrade gracefully when there is not enough memory to store them, as long as they are only being used in scan-based operations"

What are scan-based operations in the context of RDDs and which of the Transformations in Spark are scan-based operations

$\endgroup$
3
$\begingroup$

Scan based operations are basically all the operations that require evaluating the predicate on an RDD.

In other terms, each time your create an RDD or a DataFrame in which you need to compute a predicate like performing a filter, map on a case class, per example, or even explain method will be considered as a scan based operation.

To be more clear, let's review the definition of a predicate.

A predicate or a functional predicate is a logical symbol that may be applied to an object term to produce another object term.

Functional predicates are also sometimes called mappings, but that term can have other meanings as well.

Example :

// scan based transformation
rdd.filter(!_.contains("#")) // here the predicate is !_.contains("#")

// another scan based transformation
rdd.filter(myfunc) // myfunc is a boolean function

// a third also trivial scan based transformation followed by a non scan based one.
rdd.map(myfunc2) 
   .reduce(myfunc3)

If you want to understand how spark internals work, I suggest that you watch the presentation made by Databricks about the topics

|improve this answer|||||
$\endgroup$
  • $\begingroup$ That means that all Transformations in Spark are scan-based operations. Can you give some examples of Spark operations that aren't scan-based? $\endgroup$ – mbbce Oct 14 '15 at 6:08
  • $\begingroup$ I didn't say that. What I said it that each time there is a predicate to calculate, the operation is scan based. $\endgroup$ – eliasah Oct 14 '15 at 6:32
  • 1
    $\begingroup$ Ah, sorry, my bad. Can you kindly give an example of a transformation with predicate evaluation and one without it? I am (mostly) unfamiliar with Spark API atm and an example would help :) Thanks $\endgroup$ – mbbce Oct 14 '15 at 8:57
  • $\begingroup$ @MB_CE I have edited the answer. Finding a non based transformation (not actions) example is actually quite tricky and difficult. $\endgroup$ – eliasah Oct 15 '15 at 15:39
  • $\begingroup$ @eliasah what about methods like cache and unpersist? I don't think they count as transformations or actions, but they seem like an example of something that's not scan-based. $\endgroup$ – Matthew Graves Oct 15 '15 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.