New answers tagged

0

You can use the filter method on Spark's DataFrame API: df_filtered = df.filter("df.col1 = F").collect() which also supports regex pattern = r"[a-zA-Z0-9]+" df_filtered_regex = df.filter([df_filtered.c.rlike(pattern) for c in df.columns]).collect()`


0

I know this is a bit late, but I struggled with this also. This is my attempt at removing null columns from a Spark Dataframe. from pyspark.sql.functions import when, isnull colsthatarenull = df.select([(when(isnull(c), c)).alias(c) for c in df.columns]).first().asDict() namesofnullcols = {key:val for key, val in colsthatarenull.items() if val != None}....


Top 50 recent answers are included