Test dataset with categorical variable value not present in train dataset & transformer

Question

I want to replace values of a categorical variable ( named 'six' ) by the mean of my target variable ( named 'target' ).

I am fitting a transformer doing just that on a train dataset df and then transform the test dataset df2.

How do I deal with a value appearing solely in the test dataset ?

When fitted on the train dataset the transformer received no mean value of the target variable on that value.

For example :

myarray = np.array([ [ 1 , 1 , 3 , 'v' , 'a' , 'x' , 0 ] , 
                   [ 2 , 2 , 2 , 'v' , 'b' , 'y' , 1 ] ,
                   [ 4 , 5 , 1 , 'w' , 'c' , 'z' , 1 ] ,
                   [ 2 , 1 , 9 , 'w' , 'c' , 'x' , 1 ] , 
                   [ 1 , 0 , 4 , 'w' , 'b' , 'y' , 1 ] ,
                   [ 2 , 2 , 3 , 'v' , 'b' , 'z' , 0 ] ] )

colnames = [ 'one', 'two', 'three' , 'four' , 'five' , 'six' , 'target' ]

df = pd.DataFrame( myarray , columns = colnames )

myarray2 = np.array([ [ 2 , 7 , 3 , 'v' , 'a' , 'x' , 0 ] , 
                      [ 9 , 2 , 2 , 'v' , 'a' , 'y' , 0 ] ,
                      [ 4 , 5 , 1 , 'w' , 'c' , 'k' , 1 ] ]  )

colnames2 = [ 'one', 'two', 'three' , 'four' , 'five' , 'six' , 'target' ]

df2 = pd.DataFrame( myarray2 , columns = colnames2 )

df is my train dataset, df2 my test dataset.

We can see variable 'six' has the k value not existing in the train dataset.

Next :

df[ 'target' ] = df[ 'target' ].astype( 'float64' )

Next ( my homemade transformer ) :

class Cat2TargetMean( BaseEstimator , TransformerMixin ) :

    def __init__( self , col2trans , tgt_col ) :
        self._col2trans = col2trans
        self._tgt_col = tgt_col

    def fit( self, X, y = None ) :
        self._dic_col_p = {}

        for col in self._col2trans :
            p = X.groupby( col ).mean()[ self._tgt_col ]
            self._dic_col_p.update( { col : p.to_dict() } )
        return self 

    def transform( self , X , y = None ) :
        for col , dic_p in self._dic_col_p.items() : 
            X.replace( { col : dic_p } , inplace = True )  
        return X

Then :

 tsf = Cat2TargetMean( [ 'four' , 'five' , 'six' ] , 'target' )

 tsf.fit( df )

 tsf.transform( df )

 tsf.transform( df2 )

Result :

    one two three   four    five    six target
    0   2.0 7.0 3.0 0.333333    0.0 0.5 0.0
    1   9.0 2.0 2.0 0.333333    0.0 1   0.0
    2   4.0 5.0 1.0 1.000000    1.0 k   1.0

'k' value of column 'six' has not been transformed.

Answer 1

I usually replace unseen and NaN values with the global target mean.

There are also already implemented transformers for target encoding that you could use that gives you some options such as smoothing: scikit contrib - target encoder.

Answer 2

Following Simon Larsson tip ( I usually replace unseen and NaN values with the global target mean ) here is the new version of my homemade transformer :

class Cat2TargetMean( BaseEstimator , TransformerMixin ) :

def __init__( self , col2trans , tgt_col ) :
    self._col2trans = col2trans
    self._tgt_col = tgt_col

def fit( self , X , y = None ) :
    self._dfl_val = X[ self._tgt_col ].mean()
    self._dic_col_p = {}

    for col in self._col2trans :
        p = X.groupby( col ).mean()[ self._tgt_col ]
        self._dic_col_p.update( { col : p.to_dict() } )
    return self 

def transform( self , X , y = None ) :
    for col , dic_p in self._dic_col_p.items() : 
        X[ col ] = X[ col ].map( dic_p ).fillna( self._dfl_val )
    return X

I use the map function and fill the resulting NaN values by the target variable mean ( self._dfl_val ( default value ) computed in the fit method of the transformer :

self._dfl_val = X[ self._tgt_col ].mean()

Best practice should be to use the existing encoder http://contrib.scikit-learn.org/categorical-encoding/targetencoder.html#

Answer 3

You have to fit your homemade transformer on all the data:

tsf.fit( pd.concat([df, df2]))