1
If I understand what you need, I think it is this:
b = a.pivot_table(values='TOTAL_BALANCE_EUR', index=['NSFR_GROUP', 'BALANCE_GROUP'], columns='GAP', aggfunc='sum')
b
It's easier for others to help you if you make the data available to others. Just make a tiny dataframe with 10 rows for instance. Also, you can make the code a bit easier to read by ...
1
You might want to apply one-hot encoding instead. These are not really continuous features. If you consider each day of the week or month of the year a category, then you can instead treat them as categorical variables.
The year is trickier as it does not repeat itself. I would suggest to maybe instead of using the year to use a date difference: which can ...
1
The accepted answer will work, but will run df.count() for each column, which is quite taxing for a large number of columns. Calculate it once before the list comprehension and save yourself an enormous amount of time:
def drop_null_columns(df):
"""
This function drops columns containing all null values.
:param df: A PySpark ...
Only top voted, non community-wiki answers of a minimum length are eligible
