I am looking for some advice on converting my existing CNN/LSTM RNN over to a Transformer type model. This regression model takes a sliding window size of 240 rows with 33 features. It aims to predict the percent difference of the next row of data we are working with. I have fully trained this model below with ~85% accuracy which is fantastic for its use case.
Here is the existing model:
model = Sequential() model.add(Conv1D(filters=128, kernel_size=3, activation='relu', input_shape=(inp_history_size, len(features)))) model.add(BatchNormalization()) model.add(Conv1D(filters=256, kernel_size=3, activation='relu')) model.add(BatchNormalization()) model.add(Bidirectional(LSTM(256, return_sequences=True))) model.add(Bidirectional(LSTM(128, return_sequences=True))) model.add(Bidirectional(LSTM(64))) model.add(Dropout(0.3)) model.add(Dense(64, activation='elu', kernel_regularizer=l1_l2(l1=l1_reg_strength, l2=l2_reg_strength))) model.add(BatchNormalization()) model.add(Dense(1))
I have tried to craft a basic transformer model in a bid to try a different approach and possibly gain better accuracy. However, no matter what I seem to do, it stalls at ~55% accuracy and losses stop falling.
Here is the basic transformer code I have:
def transformer_encoder(inputs, head_size, num_heads, ff_dim, dropout=0): x = layers.LayerNormalization(epsilon=1e-6)(inputs) x = layers.MultiHeadAttention( key_dim=head_size, num_heads=num_heads, dropout=dropout )(x, x) x = layers.Dropout(dropout)(x) res = x + inputs x = layers.LayerNormalization(epsilon=1e-6)(res) x = layers.Conv1D(filters=ff_dim, kernel_size=1, activation="elu")(x) x = layers.Dropout(dropout)(x) x = layers.Conv1D(filters=inputs.shape[-1], kernel_size=1)(x) return x + res def build_model( input_shape, head_size, num_heads, ff_dim, num_transformer_blocks, mlp_units, dropout=0, mlp_dropout=0, ): inputs = keras.Input(shape=input_shape) x = inputs for _ in range(num_transformer_blocks): x = transformer_encoder(x, head_size, num_heads, ff_dim, dropout) x = layers.GlobalAveragePooling1D(data_format="channels_first")(x) for dim in mlp_units: x = layers.Dense(dim, activation="elu")(x) x = layers.Dropout(mlp_dropout)(x) outputs = layers.Dense(1)(x) return keras.Model(inputs, outputs) input_shape=(inp_history_size, len(features)) model = build_model( input_shape, head_size=512, num_heads=6, ff_dim=8, num_transformer_blocks=8, mlp_units=, mlp_dropout=0.2, dropout=0.1, )
I am curious as to why the transformer model stalls in learning. I have run several tests tweaking the hyperparameters but all end up with the same result.
I run a learning rate finder before every test. I hope I haven't missed anything but will update this question with further information if needed.
Can anyone lend any advice as to getting this up and running and learning well? Thank you for your help, tips and code examples in advance.