2
$\begingroup$

I want to access values of a particular column from a data sets that I've read from a csv file. The datasets are stored in pyspark RDD which I want to be converted into the DataFrame. I am using the below code :

from pyspark.sql import SQLContext
sqlc=SQLContext(sc)
df=sc.textFile(r'D:\Home\train.csv')
df=sqlc.createDataFrame(df)

but it is showing error:

Can not infer schema for type: <class 'str'>

First 2 rows of df are :

['"id","product_uid","product_title","search_term","relevance"',
 '2,100001,"Simpson Strong-Tie 12-Gauge Angle","angle bracket",3']

I think the first row is creating this problem. Moreover I want to create data frame which stores the values from 2nd row to last.(Not the first row because it will be the header). How can I achieve this ? I've searched for it but could not find any solution. Thanks in advance.

$\endgroup$
2
  • $\begingroup$ Which version of spark are you using ? $\endgroup$
    – eliasah
    Aug 7 '16 at 17:03
  • $\begingroup$ @eliasah I am using spark 2.0.0 $\endgroup$
    – Ishan
    Aug 8 '16 at 4:35
1
$\begingroup$

To read a csv file to spark dataframe you should use spark-csv. https://github.com/databricks/spark-csv

df = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('cars.csv')

How to use spark csv If you are using pyspark directly from the terminal. Instead of calling

$SPARKHOME/bin/pyspark

You have to use

$SPARKHOME/bin/pyspark --packages com.databricks:spark-csv_2.11:1.4.0 

and then use the code above.

If you are using ipython + findspark, you'll have to modify your PYSPARK_SUBMIT_ARGS (before starting ipython)

export PYSPARK_SUBMIT_ARGS=--master local[4] --packages "com.databricks:spark-csv_2.11:1.4.0" pyspark-shell 
$\endgroup$
3
  • $\begingroup$ Can you tell me how can I use them with pyspark in windows ? I am new to pyspark btw. $\endgroup$
    – Ishan
    Aug 6 '16 at 6:36
  • $\begingroup$ It's not really much different in Windows. The arguments to pyspark are still the same, you'll just have a slightly different way of setting the suggested environment variable. Possibly check this question for more, or post a separate question about running pyspark under Windows. $\endgroup$ Aug 7 '16 at 23:02
  • $\begingroup$ I am using iPython with spark, do I have to create an environment variable PYSPARK_SUBMIT_ARGS ? And whenever I start pyspark using the following command : pyspark --packages com.databricks:spark-csv_2.11:1.4.0 and then use 'sc' then it shows spark is not defined. But normally when I start pyspark, it does not show any error regarding 'sc'. $\endgroup$
    – Ishan
    Aug 8 '16 at 4:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.