I have a SQLContext data frame derived from pandas data frame consisting of several numerical columns. I want to perform multivariate statistical analysis using the pyspark.mllib.stats package. The statistics function expects a RDD of vectors. I could not convert this data frame into RDD of vectors. Is there a way to convert the data frame?
rdd = sqlCtx.createDataFrame(df_new) summary = Statistics.colStats(rdd)
I am getting df_new from
df_new = df.applymap(lambda s: dic.get(s) if s in dic else s) #df is a pandas dataframe
I am getting a PY4JJava error at the summary line. The issue is with the format of rdd.