# Problem trying to build my own sklean transformer

I build the following sklearn transformer :

class Cat2Rat( BaseEstimator , TransformerMixin ) :
def __init__( self , col2trans ) :
self._col2trans = col2trans

def fit( self, X, y = None ):
return self

def transform( self , X , y = None ) :
for col in self._col2trans :
p = X[ col ].value_counts() / X.shape[0]
dic = dict( [ ( i , p[i] ) for i in ( X[ col ].value_counts() ).index ] )
X.replace( { col : dic } , inplace = True )
return X


This transformer is replacing catagorical values by their rates.

For example :

myarray = np.array([ [ 1 , 1 , 3 , 'v' , 0 ] ,
[ 2 , 2 , 2 , 'v' , 1 ] ,
[ 4 , 5 , 1 , 'w' , 1 ] ,
[ 2 , 1 , 9 , 'w' , 1 ] ,
[ 1 , 0 , 4 , 'w' , 1 ] ] )

colnames = [ 'one', 'two', 'three' , 'four' , 'target' ]

df = pd.DataFrame( myarray , columns = colnames )


Value 'v' ( 'w' ) for column 'four' is replaced by 2/5 ( 3/5 ).

My purpose is to fit the transformer on df and apply it to another dataframe df2 :

myarray2 = np.array([ [ 2 , 7 , 3 , 'v' , 0 ] ,
[ 9 , 2 , 2 , 'v' , 0 ] ,
[ 4 , 5 , 1 , 'w' , 1 ] ]  )

colnames2 = [ 'one', 'two', 'three' , 'four' , 'target' ]

df2 = pd.DataFrame( myarray2 , columns = colnames2 )


I am doing this that way :

# Transformer instance
trsf = Cat2Rat( [ 'four' ] )

# Fitting
trsf.fit( df )

# Then applying
trsf.transform( df2 )


But the rates are those of column 'four' values of df2 not df ( on which the transformer was fitted ).

I must have been missing something on the way to properly build such a transformer.

Could someone give some clue on how to fix the transformer so that it gives proper result?

Thanks.

The definition of dic should happen inside the fit method; you're wanting the rates to be based on the dataframe passed to fit. So you'll want to set dic as another class attribute, to be referenced in the transform method.

(This kind of replacement is called target encoding, and there's an existing package to do it in sklearn.)

Considering Ben Reiniger's answer I've made some change in my sklearn transformer :

class Cat2Rat( BaseEstimator , TransformerMixin ) :
def __init__( self , col2trans ) :
self._col2trans = col2trans

def fit( self , X , y = None ):
self._dic_col_p = {}

for col in self._col2trans :
p = X[ col ].value_counts() / X.shape[0]
dic = dict( [ ( i , p[i] ) for i in ( X[ col ].value_counts() ).index ] )
self._dic_col_p.update( { col : dic } )

return self

def transform( self , X , y = None ) :

for col , dic_p in self._dic_col_p.items() :
X.replace( { col : dic_p } , inplace = True )
return X


The fit method now produce a dictionary with keys containing columns names and values containing the proportion of each columns value.

For example, self._dic_col_p contains after fitting :

{ 'four' : {'w': 0.6, 'v': 0.4} , 'five' : {'c': 0.4, 'b': 0.4, 'a': 0.2} }


where :

myarray = np.array([ [ 1 , 1 , 3 , 'v' , 'a' , 0 ] ,
[ 2 , 2 , 2 , 'v' , 'b' , 1 ] ,
[ 4 , 5 , 1 , 'w' , 'c' , 1 ] ,
[ 2 , 1 , 9 , 'w' , 'c' , 1 ] ,
[ 1 , 0 , 4 , 'w' , 'b' , 1 ] ] )

colnames = [ 'one', 'two', 'three' , 'four' , 'five' , 'target' ]

df = pd.DataFrame( myarray , columns = colnames )


Thks.