0
$\begingroup$

I am new in scala.

I have a csv file stored in hdfs. I am reading that file in scala using

  val salesdata = sc.textFile("hdfs://localhost:9000/home/jayshree/sales.csv")

Here is a small sample of data sales. a is the customerid, b-transanctionid, c-itemid, d-itemprice.

 a    b  c    d
 5  199 1   500
 33 235 1   500
 20 249 3   749
 35 36  4   757
 19 201 4   757
 17 94  5   763
 39 146 5   763
 42 162 5   763
 49 41  6   824
 3  70  6   824
 24 161 6   824
 48 216 6   824

I have to perform the following operation on it.

  1. Apply some discount on each item, on the column d(itemprice) suppose 30% of discount. The formula will be d-(30%(d))
  2. Find customer wise minimum and maximum item value after applying 30% discount to each item.

I tried to multiply 30 with the observation of column d. The problem is that the value of d as taken as string. When I am multiplying with a number in result it is show the value that many time.

I can convert it into a dataframe and do it. But I just want to know that without converting it into a dataframe can we do these operation for a RDD.

$\endgroup$
  • 1
    $\begingroup$ You are better off reading into dataframe and computing from a dataframe, to take advantage of the Catalyst optimizer. val df = spark.read.format("csv").option("header", "true").csv("/my/file.csv") $\endgroup$ – Pete Mar 23 '17 at 13:38
1
$\begingroup$

For the first you can do as follow :

val discount = salesdata.map( str => str.split(","))
                        .map( array => (array(0), array(1), array(2), array(3).toDouble) )
                        .map{ case(a, b, c, d) => (a, b, c, d-0.3*d)}

I'm not sure to understand the second, this will gives you the min and max per c-itemID

val productPrices = discount.map{ case(a, b, c, d) => (c,(d,d)) }

val minMaxPerItemRDD = productPrices.reduceByKey{ case((min1,max1),(min2,max2)) => (math.min(min1,min2), math.max(max1, max2))}

Hope that's what you need.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for your time, Kybe. I am getting this error after running the first code. "<console>:36: error: value map is not a member of (String, String, String, Double)" . I have edited my post please have a look. before the dataset was not in tabular format now I made it. I am sorry for the previous mistake. $\endgroup$ – Jaishree Rout Mar 23 '17 at 19:32
  • $\begingroup$ I'd forgot one parenthesis to the second line, it's corrected ;) $\endgroup$ – KyBe Mar 24 '17 at 14:32
  • $\begingroup$ I did it I tried that also after that only I am getting this error. "<console>:36: error: value map is not a member of (String, String, String, Double)" $\endgroup$ – Jaishree Rout Mar 24 '17 at 16:34
0
$\begingroup$

To find max and min,

var path = "filePath"    

var rdd = spark.sparkContext.textFile(path)
val headers = rdd.first()
val data_without_header=rdd.filter(line => !line.equals(headers))
data_without_header.foreach(println)

val salary_list= data_without_header.map{x => x.split(',')}.map{x=>(x(3).toInt) - (x(3).toInt)*(0.3)}

println("Max salary:" + salary_list.max())
println("Min salary:" + salary_list.min())


| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.