I'm having some trouble understanding the creation of custom transformers for Pyspark pipelines.
I am writing a custom transformer that will take the dataframe column Company and remove stray commas:
from pyspark.sql.functions import * class DFCommaDropper(Transformer): def__init__(self, *args, **kwargs): self.name = CommaDropper def transform(self,df): df = df.withColumn('Company', regexp_replace('Company',',','') return df
The above code is obviously wrong. I'm unsure what/how to initialize this and then how to use the initialized class instance in the transform function.
Thanks in advance for your help.