I have a 12 GB .csv file and I am trying to filter rows based on some value in a column under "AppName"
Doing this in Pyspark:
display(df.filter(df.AppName=="23954ec32332dfgd"))
only yields 1 row as opposed to at least 1400 rows.
Any ideas ?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this communityI have a 12 GB .csv file and I am trying to filter rows based on some value in a column under "AppName"
Doing this in Pyspark:
display(df.filter(df.AppName=="23954ec32332dfgd"))
only yields 1 row as opposed to at least 1400 rows.
Any ideas ?
Can you try this:
df1 = df.filter(df.AppName=="23954ec32332dfgd")
print df1.count()