1
$\begingroup$

I have a bi-LSTM multi-label text classification model which when training on a highly imbalanced dataset with 1000 possible labels gives a top-k (k=5) categorical accuracy of 86% and a focal loss of 0.33 when trained with about 1m rows of data. When I also used the AUC metric, it gave 96%, but I'm not comfortable reporting this to my client because they will tear it apart looking for near 100% accurate predictions.

Three questions:

  1. What is a good way of explaining the AUC to non-tech savvy people?
  2. The top-k score tells me there is room for improvement, however the AUC tells me I can chill out and boast about what I've built. From the model below, and given that I'm using transformers tokenizer to process the inputs, is there anything blindingly obvious that I could be doing better or is worth exploring further? Or should I just accept that the AUC% is very good and leave it at that?
input = Input(shape=(args.max_seq_len,))
        vocab_size = tokenizer.vocab_size

        initializer = tf.keras.initializers.HeUniform()

        model = Embedding(vocab_size, 600, input_length=args.max_seq_len, embeddings_initializer=initializer)(input)
        model = Bidirectional(LSTM(100, return_sequences=True, dropout=0.10, kernel_initializer=initializer), merge_mode='concat')(model)

        model = Flatten()(model)
        model = Dense(500, activation='relu',kernel_initializer=initializer)(model)
        model = Dropout(0.1)(model)
        model = BatchNormalization()(model)
        model = Dense(500, activation='relu', kernel_initializer=initializer)(model)
        model = Dropout(0.1)(model)
        model = BatchNormalization()(model)
        model = Dense(500, activation='relu',kernel_initializer=initializer)(model)
        model = Dropout(0.1)(model)
        model = BatchNormalization()(model)
        output = Dense(n_classes, activation='sigmoid')(model)

        model = Model(input, output)

        optimizer = tf.keras.optimizers.Adam(1e-3)
        if args.predict:
            model = load_weights(args, model)
            # Freeze layer weights during prediction mode
            model.trainable = False

        loss = tfa.losses.SigmoidFocalCrossEntropy()

        model.compile(loss=loss, optimizer=optimizer,
                      metrics=[tf.keras.metrics.TopKCategoricalAccuracy(k=5)])
  1. I read that AUC is great at dealing with imbalanced datasets, but if the model is trained no-differently depending on which metric I use, would you still consider using i.e. MLSMOTE anyway to seek further improvement?
$\endgroup$