pd is a panda module is one way of reading excel but its not available in my cluster. I want to read excel without pd module. Code1 and Code2 are two implementations i want in pyspark.
Code 1: Reading Excel
pdf = pd.read_excel(Name.xlsx)
sparkDF = sqlContext.createDataFrame(pdf)
df = sparkDF.rdd.map(list)
type(df)
Want to implement without pandas module
Code 2: gets list of strings from column colname in dataframe df
stringsList = []
columnList = list(df[colname])
for i in range(len(columnList)):
if type(columnList[i]) != float:
text = columnList[i]
stringsList.append(text.lower())
else:
stringsList.append(u'')
return stringsList
I want to implement this in pyspark.