I want to access values of a particular column from a data sets that I've read from a csv file. The datasets are stored in pyspark RDD which I want to be converted into the DataFrame. I am using the below code :
from pyspark.sql import SQLContext sqlc=SQLContext(sc) df=sc.textFile(r'D:\Home\train.csv') df=sqlc.createDataFrame(df)
but it is showing error:
Can not infer schema for type: <class 'str'>
First 2 rows of df are :
['"id","product_uid","product_title","search_term","relevance"', '2,100001,"Simpson Strong-Tie 12-Gauge Angle","angle bracket",3']
I think the first row is creating this problem. Moreover I want to create data frame which stores the values from 2nd row to last.(Not the first row because it will be the header). How can I achieve this ? I've searched for it but could not find any solution. Thanks in advance.