I have a 12 GB .csv file and I am trying to filter rows based on some value in a column under "AppName"
Doing this in Pyspark:
display(df.filter(df.AppName=="23954ec32332dfgd"))
only yields 1 row as opposed to at least 1400 rows.
Any ideas ?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Can you try this:
df1 = df.filter(df.AppName=="23954ec32332dfgd")
print df1.count()
-
$\begingroup$ If you are using python 3, then the correct way is: print(df1.count()) as stated by the docs: docs.python.org/3/whatsnew/3.0.html $\endgroup$ Jun 18, 2019 at 17:55