Using KerasClassifier for training neural network

Question

I created a simple neural network for binary spam/ham text classification using pretrained BERT transformer. The current pure-keras implementation works fine. I wanted however to plot certain metrics of the trained model, in particular the ROC curve. According to this blog post I understood that this is only be possible using KerasClassifier() from the keras.wrappers.scikit-learn package which is now deprecated and has been replaced by the scikeras package.

Thus I created a build_keras_nn() function to build my custom BERT-based neural network. I then passed this custom function to KerasClassifier() as shown in the documentation, and fitted the model using train data.

At this point I got the following error message:

ValueError: Expected 2D array, got 1D array instead: Reshape your data either using array.reshape(-1, 1) 
if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

Alright then, I thought a simple array reshape will work but I then ran into the following error:

ValueError: could not convert string to float: 'awful bio obviously hatchet job press release totally 
biased bad grammar'

So for some reason the KerasClassifier implementation doesn't allow me to directly input text, even though my preprocessing steps are included within the custom function build_keras_nn().

A full reproducible code is below:

import tensorflow_hub as hub 
import tensorflow_text as text
import tensorflow as tf 
from tensorflow.keras.layers import Input, Dropout, Dense
from tensorflow.keras.metrics import BinaryAccuracy, AUC

bert_encoder_url = "https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4"
bert_preprocessor_url = "https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3"

bert_preprocessor_model = hub.KerasLayer(bert_preprocessor_url)
bert_encoder_model = hub.KerasLayer(bert_encoder_url)

df_ = pd.read_json(spam_ham_json)   # spam_ham_json: data in JSON file as a string

X_train_, X_test_, y_train_, y_test_ = train_test_split(df_['comment_text'], df_['label'])

def build_keras_nn():

    text_input = Input(shape=(), dtype=tf.string, name="text")
    preprocessed_text = bert_preprocessor_model(text_input)
    bert_output = bert_encoder_model(preprocessed_text)
    dropout = Dropout(0.1, name='dropout')(bert_output['pooled_output'])
    classification_output = Dense(1, activation='sigmoid', name='classification_output')(dropout)

    model = tf.keras.Model(inputs=[text_input], outputs=[classification_output])

    metrics_list = [AUC(name='auc'), BinaryAccuracy(name='accuracy')]
    model.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = metrics_list)
    return model 

# Following two rows show the pure-keras implementation: this one works.
# model = build_keras_nn()
# history = model.fit(X_train_, y_train_, epochs=5);

# Now let's see the KerasClassifier
model = KerasClassifier(build_fn=build_keras_nn)
# history = model.fit(X_train_, y_train_, epochs=5);     # <-- Value Error 1
# history = model.fit(np.array(X_train_).reshape(-1,1), np.array(y_train_).reshape(-1,1), epochs=5);     # <-- Value Error 2

Data is in json format:

'{"comment_text":{"0":"problem wanted say problem trying redirect event schedule pakistan NUMBERTAG NUMBERTAG pakistan mother fucker boy want married sister ohhhh love sister boob hmmmmm yummyy","1":"get life fucking loser question ask ask katie goulet picture","2":"cum drinker hey wat nigga thought u could ban took long cuz wa busy az hell recently ill keep cumming back take word cumdrinker","3":"liar liar pant fire seriously looked contribution tennis portal page tennis page ha descussion ever please lie NUMBERTAG NUMBERTAG NUMBERTAG NUMBERTAG","4":"stop writing p nothing discus given lack bsinc education diplomacy","5":"wa fucking page one edit page","6":"question mad gay","7":"warning page nerd please leave one stay girl though pleeeeeeeeeeeeeaaaaaaaaaaaaaaaaaaaaaassssssssssssssssseeeeeeeeeeeee NUMBERTAG oneoneoneoneoneoneoneoneoneoneoenone","8":"full shit","9":"go fuck conrad black cheated thousand people pension anyone defends hm asshole apologist evil","10":"list office bearer national union student australia wp userfy userfied page located","11":"talk history scottish national party claim spying hi sentence someone belief npov claim mean someone belief npov claim","12":"section meant vice review btw magazine website writer name attached also like richardwilson NUMBERTAG even know question ninjarobotpirate wa responding happy criticise answer \\u2026 \\u2026 btw NUMBERTAG far know none editor either albanian croatian maybe airplane vision quite good think take care","13":"next time subtweet","14":"physicsyo yo yo dog","15":"self censorship tv show might might notable tv pre empted breaking news notable happens time","16":"article contains information soursed huddersfield aa street street","17":"utc onto something centrifugal force experienced mass exhibiting inertia result tiny little bullet hitting side ride merry go round rueda puthoff haisch described zero point field electronic lorenz equation coupling inertial frame reference give mass inertial reluctance rather resistance enable describe change velocity direction compare ac v dc tesla v edison NUMBERTAG NUMBERTAG NUMBERTAG june NUMBERTAG","18":"meant wa meant state either unblock create new account rendering block useless simple","19":"NUMBERTAG utc hi NUMBERTAG must mistakenly thought ian wa original member b c always viewed band definitive axeman NUMBERTAG yeah almost bought akai headrush looper year ago notorious role cab one guitarist recording settled bos loop station instead rather headrush boomerang due two reliability price issue respectively check hovercraft southpacific auburn lull kind hallucinitory guitar looping thought cab new lineup wa incredible saw NUMBERTAG skipped classic lineup NUMBERTAG compare two performance wise best NUMBERTAG NUMBERTAG NUMBERTAG may"},"label":{"0":1,"1":1,"2":1,"3":1,"4":1,"5":1,"6":1,"7":1,"8":1,"9":1,"10":0,"11":0,"12":0,"13":0,"14":0,"15":0,"16":0,"17":0,"18":0,"19":0}}'

Answer 1

I managed to solve it!

The core mistake was that scikeras doesn't accept non-numerical input. As such I needed to resort to a workaround by eliminating the text preprocessing steps from my custom build_keras_nn() and transforming them into preprocessing steps.

Custom preprocessing steps however need their own classes that inherit from scikit-learn TransformerMixin and BaseEstimator. I created two custom preprocessor classes: one to perform the BERT preprocessing task (BertPreprocessor) and the other to perform the BERT encoding task (BertEncoder). All scikit-learn-style custom transformers must implement a fit() method alongside the transform(), otherwise they are not integrable to other inbuilt scikit-learn transformers and models.

One last change was the rewrite of the input layer in build_keras_nn(). Importantly, BERT transforms words/tokens to a 768-dimensional representation, hence the shape argument of Input layer set to 768.

The final full code is as follows:

class BertPreprocessor(TransformerMixin, BaseEstimator):
    def __init__(self, preprocessor_model):
        super().__init__()
        self.preprocessor_model = preprocessor_model
    def fit(self, X, y=None):
        return self 
    def transform(self, X, y=None):
        return self.preprocessor_model(X)

class BertEncoder(TransformerMixin, BaseEstimator):
    def __init__(self, encoder_model):
        super().__init__()
        self.encoder_model = encoder_model
    def fit(self, X, y=None):
        return self
    def transform(self, X, y=None):
        return self.encoder_model(X)['pooled_output']

def build_keras_nn():
    input_layer = Input(shape=768, name='bert_pooled_output')
    dropout = Dropout(0.1, name='dropout')(input_layer)
    classification_output = Dense(1, activation='sigmoid', name='classification_output')(dropout)

    model = tf.keras.Model(inputs=[input_layer], outputs=[classification_output])

    metrics_list = [AUC(name='auc'), BinaryAccuracy(name='accuracy')]
    model.compile(optimizer = 'adam', loss='binary_crossentropy', metrics = metrics_list)
    return model 

pipe = Pipeline([
    ('preprocess', BertPreprocessor(bert_preprocessor_model)),
    ('encode', BertEncoder(bert_encoder_model)),
    ('model', KerasClassifier(build_fn=build_keras_nn, epochs=2)),
])

history = pipe.fit(X_train_, y_train_);
```