The accepted answer will work, but will run df.count() for each column, which is quite taxing for a large number of columns. Calculate it once before the list comprehension and save yourself an enormous amount of time:
This function drops columns containing all null values.
:param df: A PySpark ...
You probably mean "MapReduce" by "Hadoop". MapReduce is an older model where each operation took input from storage, computed, wrote results to storage. Of course, many real-world tasks involve lots of operations - read, filter, transform, aggregate, etc. Expressing those in MapReduce would be difficult, and what's more, every single ...
This could work. Suppose if your column name is "Marital status" and categorical,
dataset['Marital status'].replace(to_replace=v1,value= list(range(len(v1))), inplace=True)
Possible answer from here
The problem is that you're trying to pickle an object from the module where it's defined. If you move the class into a separate file and import it into your script, then it should work.
That solution isn't viable for me in an iPython notebook though. So here I some additional information from here
Python's pickle actually does ...