Skip to main content
added 50 characters in body; added 15 characters in body
Source Link

I am new in scala.

I have a csv file stored in hdfs. I am reading that file in scala using

  val salesdata = sc.textFile("hdfs://localhost:9000/home/jayshree/sales.csv")

Here is a small sample of data sales. a is the customerid, b-transanctionid, c-itemid, d-itemprice.

a b c d 5 199 1 500 33 235 1 500 20 249 3 749 35 36 4 757 19 201 4 757 17 94 5 763 39 146 5 763 42 162 5 763 49 41 6 824 3 70 6 824 24 161 6 824 48 216 6 824

 a    b  c    d
 5  199 1   500
 33 235 1   500
 20 249 3   749
 35 36  4   757
 19 201 4   757
 17 94  5   763
 39 146 5   763
 42 162 5   763
 49 41  6   824
 3  70  6   824
 24 161 6   824
 48 216 6   824

I have to perform the following operation on it.

  1. Apply some discount on each item, on the column d(itemprice) suppose 30% of discount. The formula will be d-(30%(d))
  2. Find customer wise minimum and maximum item value after applying 30% discount to each item.

I tried to multiply 30 with the observation of column d. The problem is that the value of d as taken as string. When I am multiplying with a number in result it is show the value that many time.

I can convert it into a dataframe and do it. But I just want to know that without converting it into a dataframe can we do these operation for a RDD.

I am new in scala.

I have a csv file stored in hdfs. I am reading that file in scala using

  val salesdata = sc.textFile("hdfs://localhost:9000/home/jayshree/sales.csv")

Here is a small sample of data sales. a is the customerid, b-transanctionid, c-itemid, d-itemprice.

a b c d 5 199 1 500 33 235 1 500 20 249 3 749 35 36 4 757 19 201 4 757 17 94 5 763 39 146 5 763 42 162 5 763 49 41 6 824 3 70 6 824 24 161 6 824 48 216 6 824

I have to perform the following operation on it.

  1. Apply some discount on each item, on the column d(itemprice) suppose 30% of discount. The formula will be d-(30%(d))
  2. Find customer wise minimum and maximum item value after applying 30% discount to each item.

I tried to multiply 30 with the observation of column d. The problem is that the value of d as taken as string. When I am multiplying with a number in result it is show the value that many time.

I can convert it into a dataframe and do it. But I just want to know that without converting it into a dataframe can we do these operation for a RDD.

I am new in scala.

I have a csv file stored in hdfs. I am reading that file in scala using

  val salesdata = sc.textFile("hdfs://localhost:9000/home/jayshree/sales.csv")

Here is a small sample of data sales. a is the customerid, b-transanctionid, c-itemid, d-itemprice.

 a    b  c    d
 5  199 1   500
 33 235 1   500
 20 249 3   749
 35 36  4   757
 19 201 4   757
 17 94  5   763
 39 146 5   763
 42 162 5   763
 49 41  6   824
 3  70  6   824
 24 161 6   824
 48 216 6   824

I have to perform the following operation on it.

  1. Apply some discount on each item, on the column d(itemprice) suppose 30% of discount. The formula will be d-(30%(d))
  2. Find customer wise minimum and maximum item value after applying 30% discount to each item.

I tried to multiply 30 with the observation of column d. The problem is that the value of d as taken as string. When I am multiplying with a number in result it is show the value that many time.

I can convert it into a dataframe and do it. But I just want to know that without converting it into a dataframe can we do these operation for a RDD.

Source Link

Scala RDD operation

I am new in scala.

I have a csv file stored in hdfs. I am reading that file in scala using

  val salesdata = sc.textFile("hdfs://localhost:9000/home/jayshree/sales.csv")

Here is a small sample of data sales. a is the customerid, b-transanctionid, c-itemid, d-itemprice.

a b c d 5 199 1 500 33 235 1 500 20 249 3 749 35 36 4 757 19 201 4 757 17 94 5 763 39 146 5 763 42 162 5 763 49 41 6 824 3 70 6 824 24 161 6 824 48 216 6 824

I have to perform the following operation on it.

  1. Apply some discount on each item, on the column d(itemprice) suppose 30% of discount. The formula will be d-(30%(d))
  2. Find customer wise minimum and maximum item value after applying 30% discount to each item.

I tried to multiply 30 with the observation of column d. The problem is that the value of d as taken as string. When I am multiplying with a number in result it is show the value that many time.

I can convert it into a dataframe and do it. But I just want to know that without converting it into a dataframe can we do these operation for a RDD.