Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Currently I'm using pyspark to make my df from a csv. However when I take the data in, it puts each element on a new line. Is there any way to keep the elements separate, and keep them on the same line?
#Loads the data in the textfile
df = sc.textFile("data.csv").map(lambda line: line.split(","))
df.take(1)
[['2',
'20200118',
'0',
'AE',
'ABV',
'AUH',
'20200416',
'0',
You actually have a matrix, all items are grouped by row.
A very small example:
csv:
a,b
c,d
Will become:
[['a','b'],['c','d']]
The formatting of the output is indeed misleading.
df.take(1)as it printed is also the actual structure of your data, meaning that it is actually not including the new lines in it i.e:[['2', '20200118', '0', 'AE', 'ABV', 'AUH', '20200416', '0', ...], [row_2...]]Did you try assigning it to a df to see if the new lines are of elements or of the rows? $\endgroup$ – Romid Jan 21 '20 at 9:23