0
$\begingroup$

In the example given for the max function for PySpark:

Pyspark

>>> rdd = sc.parallelize([1.0, 5.0, 43.0, 10.0])
>>> rdd.max()
43.0
>>> rdd.max(key=str)
5.0

Q1. How does it get evaluated to 5.0 when key=str ? Is it based on conversion to character/string type ?

Q2. Is there any value the parameter "key" can take ?

I also found the function definition of "max" at this location

https://github.com/adobe-research/spark-gpu/blob/master/src/rdd.py

def max(self, key=None):
        """
        Find the maximum item in this RDD.
        :param key: A function used to generate key for comparing
        >>> rdd = sc.parallelize([1.0, 5.0, 43.0, 10.0])
        >>> rdd.max()
        43.0
        >>> rdd.max(key=str)
        5.0
        """
        if key is None:
            return self.reduce(max)
        return self.reduce(lambda a, b: max(a, b, key=key))
$\endgroup$
1
  • $\begingroup$ rdd.reduce(lambda a, b: a if (a[1] > b[1]) else b) $\endgroup$
    – Shuai.Z
    Nov 17, 2016 at 1:29

2 Answers 2

3
$\begingroup$

You pass a function to the key parameter that it will virtually map your rows on to check for the maximum value. In this case you pass the str function which converts your floats to strings. Since '5.0' > '14.0' due to the nature of string comparisons, this is returned. What is usually a more likely use is using the key parameter as follows:

test = sc.parallelize([(1, 2), (4,3), (2,4)])
test.max(key = lambda x: -x[1])
(1,2)

Because of the - we sort descending and the x[1] means we use the second entry in our tuples as the key.

$\endgroup$
0
$\begingroup$

Compares the value based on position in number

The numbers are compared from highest to lowest. Here 9>8 where 8 comes from 89:

x = spark.sparkContext.parallelize([1,2,3,4,5,6,7,89,7,33,9])
x.max()           #Output 89
x.max(key=str)    #Output 9
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.