I have a sequence classification model featuring CustomELMo Embeddings layer + BiLSTM + Fully Connected layer.

I've found two custom metrics for sparse_categorical_accuracy, but can't wrap my head around exactly what they are doing, what the difference are, and whether one is one better than the other?

I'm quite sure the shape of predictions going into both methods are: y_pred = (number of sents, sentence length (sentences are padded to max_len), number of tags), y_true = (number of sents, sentence length, 1)

Method 1: Commented by jerrywind on this page

def custom_sparse_categorical_accuracy(y_true, y_pred):
    flatten_y_true = K.cast( K.reshape(y_true,(-1,1) ), K.floatx())
    flatten_y_pred = K.cast(K.reshape(y_pred, (-1, y_pred.shape[-1])), K.floatx())
    y_pred_labels = K.cast(K.argmax(flatten_y_pred, axis=-1), K.floatx())
    return K.cast(K.equal(flatten_y_true,y_pred_labels), K.floatx())

Method 2: commented by dilshatu on this page:

def custom_sparse_categorical_accuracy(y_true, y_pred):
    return K.cast(K.equal(K.max(y_true, axis=-1),                K.cast(K.argmax(y_pred, axis=-1), K.floatx())), K.floatx())

When I try either method on a tiny sample of my data and view the model history, the train and validation accuracies for that method aren't identical but very close (sometimes the same values).

The sparse_categorical_accuracy scores after model.evaulate are quite different: 89.69% vs 93.49% for method 1 vs method 2.

The F1-scores after model.predict however are close: 19.0% vs 19.4% for method 1 vs method 2.

My model is constructed as follows:

def build_model():
    input = layers.Input(shape=(1,), dtype=tf.string)
    # Custom ELMo embeddings layer
    model = ElmoEmbeddingLayer(name='ElmoEmbeddingLayer')(input)
    model = Bidirectional(LSTM(units=512, return_sequences=True))(model)
    out = TimeDistributed(Dense(num_tags, activation='softmax'))(model)
    model = Model(inputs=input, outputs=out)
    model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy', metrics=[custom_sparse_categorical_accuracy])
    return model

model = build_model()
epochs = 10
batch_size = 32
history = model.fit(X_train_sents,
                validation_data=(X_valid_sents, y_valid),
                batch_size=batch_size, epochs=epochs)

My task is semantic slot filling (a kind of named entity recognition). I'm not worried about the bad scores because I'm using a tiny sample and just 10 epochs.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.