I have a spreadsheet of banking information and one of the goals is to find out the failure rate of an advertising campaign. I think I need to get a count of the total number of entries vs. the total number of non-subscribers. The column for subscribers is a simple "yes" or "no" instead of an integer.
Of course, getting the total number of entries was easy:
scala> val input = sc.textFile("project_1_data.csv") input: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD at textFile at <console>:27 scala> input.count() res0: Long = 45211
Next, I filter for "no" in the subscriber column (called "y"):
scala> val sub = bankDF.filter($"y" === "no") sub: org.apache.spark.sql.DataFrame = [age: int, job: string, marital: string, education: string, default: string, balance: int, housing: string, loan: string, contact: string, day: string, month: string, duration: int, campaign: int, pdays: string, previous: int, poutcome: string, y: string] scala> sub.count() res2: Long = 39922
So now I have two numbers to work with, but they're the results of two different operations. How can I work out a percentage of these two numbers in a single pass?
Also, if I may add a second question, is there maybe a reference page on how to do averages, medians, means, etc. - so I don't have to ask you guys questions every time? Thanks in advance for any assistance!