3
$\begingroup$

I build the following sklearn transformer :

class Cat2Rat( BaseEstimator , TransformerMixin ) :
   def __init__( self , col2trans ) :
      self._col2trans = col2trans

   def fit( self, X, y = None ):
      return self 

   def transform( self , X , y = None ) :
      for col in self._col2trans : 
         p = X[ col ].value_counts() / X.shape[0]
         dic = dict( [ ( i , p[i] ) for i in ( X[ col ].value_counts() ).index ] )
         X.replace( { col : dic } , inplace = True )
      return X

This transformer is replacing catagorical values by their rates.

For example :

myarray = np.array([ [ 1 , 1 , 3 , 'v' , 0 ] , 
                 [ 2 , 2 , 2 , 'v' , 1 ] ,
                 [ 4 , 5 , 1 , 'w' , 1 ] ,
                 [ 2 , 1 , 9 , 'w' , 1 ] , 
                 [ 1 , 0 , 4 , 'w' , 1 ] ] )

colnames = [ 'one', 'two', 'three' , 'four' , 'target' ]

df = pd.DataFrame( myarray , columns = colnames )

Value 'v' ( 'w' ) for column 'four' is replaced by 2/5 ( 3/5 ).

My purpose is to fit the transformer on df and apply it to another dataframe df2 :

myarray2 = np.array([ [ 2 , 7 , 3 , 'v' , 0 ] , 
                    [ 9 , 2 , 2 , 'v' , 0 ] ,
                    [ 4 , 5 , 1 , 'w' , 1 ] ]  )

colnames2 = [ 'one', 'two', 'three' , 'four' , 'target' ]

df2 = pd.DataFrame( myarray2 , columns = colnames2 )

I am doing this that way :

# Transformer instance
trsf = Cat2Rat( [ 'four' ] )

# Fitting
trsf.fit( df )

# Then applying
trsf.transform( df2 )

But the rates are those of column 'four' values of df2 not df ( on which the transformer was fitted ).

I must have been missing something on the way to properly build such a transformer.

Could someone give some clue on how to fix the transformer so that it gives proper result?

Thanks.

$\endgroup$
1
$\begingroup$

The definition of dic should happen inside the fit method; you're wanting the rates to be based on the dataframe passed to fit. So you'll want to set dic as another class attribute, to be referenced in the transform method.

(This kind of replacement is called target encoding, and there's an existing package to do it in sklearn.)

|improve this answer|||||
$\endgroup$
0
$\begingroup$

Considering Ben Reiniger's answer I've made some change in my sklearn transformer :

class Cat2Rat( BaseEstimator , TransformerMixin ) :
   def __init__( self , col2trans ) :
       self._col2trans = col2trans


   def fit( self , X , y = None ):
       self._dic_col_p = {}

       for col in self._col2trans :
           p = X[ col ].value_counts() / X.shape[0]
           dic = dict( [ ( i , p[i] ) for i in ( X[ col ].value_counts() ).index ] )
           self._dic_col_p.update( { col : dic } )

       return self


   def transform( self , X , y = None ) :

       for col , dic_p in self._dic_col_p.items() : 
           X.replace( { col : dic_p } , inplace = True )
       return X

The fit method now produce a dictionary with keys containing columns names and values containing the proportion of each columns value.

For example, self._dic_col_p contains after fitting :

{ 'four' : {'w': 0.6, 'v': 0.4} , 'five' : {'c': 0.4, 'b': 0.4, 'a': 0.2} }

where :

myarray = np.array([ [ 1 , 1 , 3 , 'v' , 'a' , 0 ] , 
                   [ 2 , 2 , 2 , 'v' , 'b' , 1 ] ,
                   [ 4 , 5 , 1 , 'w' , 'c' , 1 ] ,
                   [ 2 , 1 , 9 , 'w' , 'c' , 1 ] , 
                   [ 1 , 0 , 4 , 'w' , 'b' , 1 ] ] )

colnames = [ 'one', 'two', 'three' , 'four' , 'five' , 'target' ]

df = pd.DataFrame( myarray , columns = colnames )

Thks.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.