1

If I understand what you need, I think it is this: b = a.pivot_table(values='TOTAL_BALANCE_EUR', index=['NSFR_GROUP', 'BALANCE_GROUP'], columns='GAP', aggfunc='sum') b It's easier for others to help you if you make the data available to others. Just make a tiny dataframe with 10 rows for instance. Also, you can make the code a bit easier to read by ...


1

You might want to apply one-hot encoding instead. These are not really continuous features. If you consider each day of the week or month of the year a category, then you can instead treat them as categorical variables. The year is trickier as it does not repeat itself. I would suggest to maybe instead of using the year to use a date difference: which can ...


1

The accepted answer will work, but will run df.count() for each column, which is quite taxing for a large number of columns. Calculate it once before the list comprehension and save yourself an enormous amount of time: def drop_null_columns(df): """ This function drops columns containing all null values. :param df: A PySpark ...


Only top voted, non community-wiki answers of a minimum length are eligible