0
$\begingroup$

I am building a denoising autoencoder (DAE) to denoise respiratory signals. I pass through the model both noisy and clean versions of the signal (in frame sizes as multiples of 1024).

I've set up my model up as follows:

class NoiseReducer(tf.keras.Model):
    
    def __init__(self):
        super().__init__()
        
        self.encoder = tf.keras.Sequential([
#             Input(shape=(window_size, 1)),
            Masking(mask_value=np.nan, input_shape=(window_size, 1)),
            Conv1D(filters=128, kernel_size=32, strides=1, kernel_constraint=max_norm(max_norm_value), padding='same', kernel_initializer='glorot_normal', activation='elu'),
            Dense(128, activation='elu'),
            Conv1D(filters=32, kernel_size=16, strides=1, kernel_constraint=max_norm(max_norm_value), padding='same', kernel_initializer='glorot_normal', activation='elu'),
            Conv1D(filters=16, kernel_size=8, strides=1, kernel_constraint=max_norm(max_norm_value), padding='same', kernel_initializer='glorot_normal', activation='elu')
        ])
        
        self.decoder = tf.keras.Sequential([
            Conv1DTranspose(filters=16, kernel_size=8, strides=1, kernel_constraint=max_norm(max_norm_value), padding='same', kernel_initializer='glorot_normal', activation='elu')
            Conv1DTranspose(filters=32, kernel_size=16, strides=1, kernel_constraint=max_norm(max_norm_value), padding='same', kernel_initializer='glorot_normal', activation='elu'),
            Dense(128, activation='elu'),
            Conv1DTranspose(filters=128, kernel_size=32, strides=1, kernel_constraint=max_norm(max_norm_value), padding='same', kernel_initializer='glorot_normal', activation='elu'),
            Conv1D(filters=1, kernel_size=2, strides=1, kernel_constraint=max_norm(max_norm_value), padding='same', activation='sigmoid')
        ])
        
    def call(self, x): 
        encoded = self.encoder(x) 
        decoded = self.decoder(encoded)
        return decoded

dae = NoiseReducer()

adam_optimizer=tf.keras.optimizers.Adam(
    learning_rate=learning_rate, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
sgd_optimizer = tf.keras.optimizers.SGD(learning_rate=learning_rate)
dae.compile(optimizer=sgd_optimizer, loss='mean_squared_error', metrics='accuracy')

history = dae.fit(X_noisy_train, 
        X_clean_train,
        epochs=epochs,
        batch_size=batch_size,
        shuffle=False,
        validation_split=0.3,
        callbacks=[tb_callback]
)

RESULTS:

 13/13 [==============================] - 16s 1s/step - loss: 0.2185 - accuracy: 0.8272 - val_loss: 0.2143 - val_accuracy: 0.8288
    Epoch 2/100
    13/13 [==============================] - 12s 898ms/step - loss: 0.2120 - accuracy: 0.8272 - val_loss: 0.2082 - val_accuracy: 0.8288
    Epoch 3/100
    13/13 [==============================] - 12s 908ms/step - loss: 0.2057 - accuracy: 0.8272 - val_loss: 0.2017 - val_accuracy: 0.8288
    Epoch 4/100
    13/13 [==============================] - 12s 906ms/step - loss: 0.1997 - accuracy: 0.8272 - val_loss: 0.1956 - val_accuracy: 0.8288
    Epoch 5/100
    13/13 [==============================] - 12s 907ms/step - loss: 0.1938 - accuracy: 0.8272 - val_loss: 0.1898 - val_accuracy: 0.8288

When running the model, the accuracy and validation accuracy is stuck at around 0.827 for both and does not change at all throughout the epochs (100 in total) suggesting that the model isn't learning anything. The MSE is however descreasing with epochs.

For my datasets I have set any nan values to 0

In terms of solutions I have implemented the following changes to my model but to no success:

  • Increased filter length of conv1D layers
  • Tested different learning rates for both SGD and Adam
  • Test with ELU (Exponential Linear Unit) activation instead of RELU
  • Use more int. dense layers with more neurons.
  • Using glorot (commonly known as Xavier) initializer
  • Use SGD instead of Adam
  • Change window size with different multiples of 1024

None of these seem to change the accuracy. After model completion and reconstructing the signal (from the noisy) I get a straight line cutting through 0.345 illustrating that the model has not learnt anything and can not reconstruct the signal.

What other strategies/alleys should I explore around this?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.