1
$\begingroup$

I have a basic sequential neural network built with TensorFlow.

model = tf.keras.Sequential([
    Dense(16, activation='relu', input_shape=(X_train.shape[1],), kernel_regularizer=l1_l2(0.001, 0.001)),
    Dropout(0.3),
    BatchNormalization(),
    Dense(64, activation='relu', kernel_regularizer=l1_l2(0.001, 0.001)),
    Dropout(0.3),
    BatchNormalization(),
    Dense(64, activation='relu'),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['binary_accuracy', 'AUC'])

I train on 12,000 samples with are split evenly. 6000 are category == 1 and 6000 are category == 0. Currently my network treats each category equally. It is equally likely to correctly/wrongly categorise both categories (about 92% and 93% accuracy). However, in my application I need category 1 to be correctly identified >99% of the time. category 0 accuracy can be reduced as low as 84% to achieve this.

How could I go about doing this?

$\endgroup$
1
  • 1
    $\begingroup$ What exactly do you mean by $99\%$ and $84\%$ accuracy for each category? Do you mean that, of the members of each category, $99\%$ and $84\%$, respectively, must be identified as such, or of the cases identified as belonging to each category, $99\%$ and $84\%$ must really belong to that category? These are not quite the same. // Note that your data might not support the performance you want. Nothing in the question supports such performance either being possible or impossible. // Remember that your network makes predictions on a continuum, rather than making hard classifications. $\endgroup$
    – Dave
    Commented Mar 15, 2023 at 10:17

1 Answer 1

1
$\begingroup$

Adjusting your classification threshold is probably the easiest solution to try first. By default, you're likely classifying probabilities above 0.5 as class 1. Lower threshold would mean more entries classified as 1 at the expense of more false positives.

First of all, you should define what you mean by category accuracy. Is that recall (the share of true positives among actual positives) or precision (the share of true positives among predicted positives) or something else entirely?

Then, you can apply instruments such as sklearn precision/recall curve to select your optimal threshold.

$\endgroup$
2
  • $\begingroup$ Sorry yes by accuracy I meant recall. I will look into optimising the probability classification threshold. Thank you. $\endgroup$ Commented Mar 15, 2023 at 10:35
  • $\begingroup$ @ChandlerKenworthy Note that the predictions made by this model might not support the requirements you have. That is, there might not be any threshold giving the performance you demand. (In fact, the data you have might not support such performance, regardless of the model. It might just be the case that you are missing determinants of the outcome.) $\endgroup$
    – Dave
    Commented Mar 15, 2023 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.