I'm having some trouble understanding the creation of custom transformers for Pyspark pipelines.
I am writing a custom transformer that will take the dataframe column Company and remove stray commas:
from pyspark.sql.functions import *
class DFCommaDropper(Transformer):
def__init__(self, *args, **kwargs):
self.name = CommaDropper
def transform(self,df):
df = df.withColumn('Company', regexp_replace('Company',',','')
return df
The above code is obviously wrong. I'm unsure what/how to initialize this and then how to use the initialized class instance in the transform function.
Thanks in advance for your help.