0
$\begingroup$

I am building a simple BERT model for text classification, using the tensorflow hub.

import tensorflow as tf
import tensorflow_hub as tf_hub

bert_preprocess = tf_hub.KerasLayer("https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3")
bert_encoder = tf_hub.KerasLayer("https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4")


text_input = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')
preprocessed_text = bert_preprocess(text_input)
encoded_input = bert_encoder(preprocessed_text)

l1 = tf.keras.layers.Dropout(0.3, name="dropout1")(encoded_input['pooled_output'])
l2 = tf.keras.layers.Dense(1, activation='sigmoid', name="output")(l1)

model = tf.keras.Model(inputs=[text_input], outputs = [l2])

model.summary()

Upon analyzing the output of the bert_preprocess step, I noticed that they are arrays of length 128. My texts are much shorter on average than 128 tokens and as such, my intention would be to decrease this length parameter, so that the preprocessing yields, say, arrays of length 40 only. However, I cannot figure out how to pass this max_length or output_shape parameter to the bert_preprocess.

Printed model summary:

__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 text (InputLayer)              [(None,)]            0           []                               
                                                                                                  
 keras_layer_16 (KerasLayer)    {'input_word_ids':   0           ['text[0][0]']                   
                                (None, 128),                                                      
                                 'input_type_ids':                                                
                                (None, 128),                                                      
                                 'input_mask': (Non                                               
                                e, 128)}                                                          
                                                                                                  
 keras_layer_17 (KerasLayer)    {'sequence_output':  109482241   ['keras_layer_16[0][0]',         
                                 (None, 128, 768),                'keras_layer_16[0][1]',         
                                 'default': (None,                'keras_layer_16[0][2]']         
                                768),                                                             
                                 'encoder_outputs':                                               
                                 [(None, 128, 768),                                               
                                 (None, 128, 768),                                                
                                 (None, 128, 768),                                                
                                 (None, 128, 768),                                                
                                 (None, 128, 768),                                                
                                 (None, 128, 768),                                                
                                 (None, 128, 768),                                                
...
Total params: 109,483,010
Trainable params: 769
Non-trainable params: 109,482,241

Checking the documentation, I found there is a output_shape argument for tf_hub.KerasLayer, so I tried passing the following arguments:

bert_preprocess = tf_hub.KerasLayer("https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3", output_shape=(64,))

bert_preprocess = tf_hub.KerasLayer("https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3", output_shape=[64])

However, in both of these cases, the following line throws an error:

bert_preprocess("we have a very sunny day today don't you think so?")

Error:

ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_23952\4048288771.py in <module>
----> 1 bert_preprocess("we have a very sunny day today don't you think so?")

~\AppData\Roaming\Python\Python37\site-packages\keras\utils\traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

c:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_hub\keras_layer.py in call(self, inputs, training)
    237       result = smart_cond.smart_cond(training,
    238                                      lambda: f(training=True),
--> 239                                      lambda: f(training=False))
    240 
    241     # Unwrap dicts returned by signatures.

c:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_hub\keras_layer.py in <lambda>()
    237       result = smart_cond.smart_cond(training,
    238                                      lambda: f(training=True),
--> 239                                      lambda: f(training=False))
    240 
    241     # Unwrap dicts returned by signatures.
...
  Keyword arguments: {}

Call arguments received:
  • inputs="we have a very sunny day today don't you think so?"
  • training=False
```
$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy