2
$\begingroup$

I am currently trying to implement a model which can detect a specific sequence according to the training data which looks like the following:

x_train = [...] # shape: (timesteps, features)
y_train = [[0], [0], [0], [0], [1]]  # shape: (timesteps, 1)

The model:

    rnn = GRU(units=32,
        activation="tanh", recurrent_activation="sigmoid",
        recurrent_dropout=0.0, stateful=True,
        return_sequences=True,
        unroll=False, use_bias=True, reset_after=True,
        dropout=0.2)

    dense_pre = Dense(units=32, activation=relu)
    dense_post = Dense(units=1, activation="sigmoid")

    # Input
    input_shape = (39, 20)
    inputs = Input(
        shape=input_shape,
        batch_size=batch_size,
        dtype=tf.float32
    )

    # Feed input through the network

    outputs = dense_pre(inputs)
    outputs = rnn(outputs)
    outputs = dense_post(outputs)

    outputs = tf.reshape(outputs, tf.shape(outputs)[:-1])

    adam_opt = Adam()
    metrics = ["accuracy"]

    model = Model(inputs=inputs, outputs=outputs)
    model.compile(optimizer=adam_opt,
        loss=binary_crossentropy,
        metrics=metrics)

Through stateful=True the model should learn to support "streaming" capabilties, such that by the time we are doing inference, we can feed the network data chunkwise without manually resetting the state.

With return_sequences=True we can produce a probability for each timestep.

The problem I am currently facing is that it somehow converges too fast while reaching low loss very quickly (I would assume that the low loss would result into the stagnation of the learning process). I am wondering whether this approach is somehow sub-optimal or whether I am missing something very crucial here.

I also have the suspicion that the loss function/y_true might not be optimal here.

train-acc train-loss

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.