0
$\begingroup$

Currently I'm using pyspark to make my df from a csv. However when I take the data in, it puts each element on a new line. Is there any way to keep the elements separate, and keep them on the same line?

#Loads the data in the textfile
df = sc.textFile("data.csv").map(lambda line: line.split(","))
df.take(1)

[['2',

  '20200118',

  '0',

  'AE',

  'ABV',

  'AUH',

  '20200416',

  '0',
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I'm not sure that the output of the df.take(1) as it printed is also the actual structure of your data, meaning that it is actually not including the new lines in it i.e: [['2', '20200118', '0', 'AE', 'ABV', 'AUH', '20200416', '0', ...], [row_2...]] Did you try assigning it to a df to see if the new lines are of elements or of the rows? $\endgroup$
    – Romid
    Jan 21 '20 at 9:23
  • $\begingroup$ Have you tried to use the pandas.read_csv? $\endgroup$
    – Andrea Moro
    Feb 20 '20 at 15:06
1
$\begingroup$

You actually have a matrix, all items are grouped by row.

A very small example:

csv:
a,b
c,d

Will become:

[['a','b'],['c','d']]

The formatting of the output is indeed misleading.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.