1
$\begingroup$

I have a basic sequential neural network built with TensorFlow.

model = tf.keras.Sequential([
    Dense(16, activation='relu', input_shape=(X_train.shape[1],), kernel_regularizer=l1_l2(0.001, 0.001)),
    Dropout(0.3),
    BatchNormalization(),
    Dense(64, activation='relu', kernel_regularizer=l1_l2(0.001, 0.001)),
    Dropout(0.3),
    BatchNormalization(),
    Dense(64, activation='relu'),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['binary_accuracy', 'AUC'])

I train on 12,000 samples with are split evenly. 6000 are category == 1 and 6000 are category == 0. Currently my network treats each category equally. It is equally likely to correctly/wrongly categorise both categories (about 92% and 93% accuracy). However, in my application I need category 1 to be correctly identified >99% of the time. category 0 accuracy can be reduced as low as 84% to achieve this.

How could I go about doing this?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ What exactly do you mean by $99\%$ and $84\%$ accuracy for each category? Do you mean that, of the members of each category, $99\%$ and $84\%$, respectively, must be identified as such, or of the cases identified as belonging to each category, $99\%$ and $84\%$ must really belong to that category? These are not quite the same. // Note that your data might not support the performance you want. Nothing in the question supports such performance either being possible or impossible. // Remember that your network makes predictions on a continuum, rather than making hard classifications. $\endgroup$
    – Dave
    Mar 15 at 10:17

1 Answer 1

Reset to default
1
$\begingroup$

Adjusting your classification threshold is probably the easiest solution to try first. By default, you're likely classifying probabilities above 0.5 as class 1. Lower threshold would mean more entries classified as 1 at the expense of more false positives.

First of all, you should define what you mean by category accuracy. Is that recall (the share of true positives among actual positives) or precision (the share of true positives among predicted positives) or something else entirely?

Then, you can apply instruments such as sklearn precision/recall curve to select your optimal threshold.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Sorry yes by accuracy I meant recall. I will look into optimising the probability classification threshold. Thank you. $\endgroup$
    – Chandler Kenworthy
    Mar 15 at 10:35
  • $\begingroup$ @ChandlerKenworthy Note that the predictions made by this model might not support the requirements you have. That is, there might not be any threshold giving the performance you demand. (In fact, the data you have might not support such performance, regardless of the model. It might just be the case that you are missing determinants of the outcome.) $\endgroup$
    – Dave
    Mar 15 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.