1
$\begingroup$

Hello ive been training a CNN with keras. A binnary clasificator where it says if a depth image has a manhole or not. Ive labeled manually the datasets with 0 (no manhole) and 1(it has a manhole). I have 2 datasets 1 with 45k images to train the CNN and one with 26k images to test the CNN.

Both datasets are unbalanced double of negatives images than positives.

This is the code:

# dimensions of our images.
img_width, img_height = 80, 60
n_positives_img, n_negatives_img = 17874, 26308 
n_total_img = 44182

#Labeled arrays for datasets
arrayceros = np.zeros(n_negatives_img)
arrayunos = np.ones(n_positives_img)

#Reshaping of datasets to convert separate them
arraynegativos= ds_negatives.reshape(( n_negatives_img, img_height, img_width,1))
arraypositivos= ds_positives.reshape((n_positives_img, img_height, img_width,1))

#Labeling datasets with the arrays
ds_negatives_target = tf.data.Dataset.from_tensor_slices((arraynegativos, arrayceros))
ds_positives_target = tf.data.Dataset.from_tensor_slices((arraypositivos, arrayunos))

#Concatenate 2 datasets and shuffle them
ds_concatenate = ds_negatives_target.concatenate(ds_positives_target)
datasetfinal = ds_concatenate.shuffle(n_total_img)

Then i have the same for the second dataset for testing.

#Adding batch dimension to datasets 4dim
valid_ds = datasetfinal2.batch(12)
train_ds = datasetfinal.batch(12)

#Defining model
model = Sequential()
model.add(Conv2D(5, kernel_size=(5, 5),activation='relu',input_shape=(60,80,1),padding='same'))
model.add(BatchNormalization())
model.add(MaxPooling2D((5, 5),padding='same'))
model.add(Dropout(0.3))
model.add(Conv2D(5, (5, 5), activation='relu',padding='same'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))
model.add(Dropout(0.3))
model.add(Conv2D(5, (5, 5), activation='relu',padding='same'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))
model.add(Dropout(0.3))
model.add(Conv2D(5, (5, 5), activation='relu',padding='same'))
model.add(BatchNormalization())
model.add(Flatten())
model.add(Dropout(0.5))
model.add(Dense(100, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(50, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(1, activation='sigmoid'))

#Compiling model
model.summary()

initial_learning_rate = 0.001
lr_schedule = keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate, decay_steps=100000, decay_rate=0.96, staircase=True
)
model.compile(
    loss="binary_crossentropy",
    optimizer=keras.optimizers.Adam(learning_rate=lr_schedule),
    metrics=["acc"],
)

# Define callbacks.
checkpoint_cb = keras.callbacks.ModelCheckpoint(
    "2d_image_classification.h5", save_best_only=True
)
early_stopping_cb = keras.callbacks.EarlyStopping(monitor="val_acc", patience=15)

#Fitting the model
history= model.fit(train_ds, validation_data=valid_ds, batch_size=100, epochs=5,callbacks=[checkpoint_cb, early_stopping_cb])

This gives me 99% of acc in train dataset and 95% in test dataset. But when i do this it gives me 60% precision for negatives images and 45% for positives:

#Get the real labels of valid dataset
valid_labels = list(valid_ds.flat_map(lambda x, y: tf.data.Dataset.from_tensor_slices((x, y))).as_numpy_iterator())
valid_labels = [y for x, y in valid_labels]

y_pred = model.predict(valid_ds)
y_pred = (y_pred > 0.5).astype(float)

from sklearn.metrics import classification_report
print(classification_report(valid_labels, y_pred))

Why this? I have printed both predicted labels and true labels and it look likes its random. It has no sense.

https://colab.research.google.com/drive/1bhrntDItqoeT0KLb-aKp0W8cV6LOQOtP?usp=sharing

If u need more information, just ask me.

Thanks!!!!

$\endgroup$

2 Answers 2

2
$\begingroup$

Accuracy is not a good metric when you have an unbalanced Dataset. Imagine a binary classification with a dataset composed of 90% of '0' and 10% of '1'.

If you make a model that always predict '0', (so which is useless, because your goal is to identify ones), it'll have a 90% accuracy.

Since you obtain 99% accuracy, I believe you trained your model in a goal to maximize this metric. With what I explained before, you can understand this is a bad idea.

Precision and Recall (you're quoting in your question) are already way better idea to look to understand your model's performance and train / tune it. You can use one of the metric such as AUC (independant from dataset balancement), way better than accuracy in your case, to compare your models.

$\endgroup$
8
  • $\begingroup$ I get this and i know getting confusion matrix and recall result that my cnn is bad trained. But that was the question where is the problem in my cnn? why i get this bad results? recall result are: for label 0 56% and por label 1 45%. Is my cnn overfitting or what? Thanks!! $\endgroup$ Oct 4, 2021 at 10:08
  • $\begingroup$ BTW: with auc i get 0.501 of score. $\endgroup$ Oct 4, 2021 at 10:16
  • 1
    $\begingroup$ AUC = 0.5 means your model is a random classifier (really bad performance), so your AUC shows your model is really bad. It might be what I said in my answer : You focus on increasing accuracy in a biased problem. (also are you sure about how you calculate AUC and Accuracy ? If you're unbalance is only 2/3 1/3, it feels really strange to me to have 99% accuracy with 0.5 AUC...) $\endgroup$
    – Adept
    Oct 4, 2021 at 11:28
  • $\begingroup$ I also got the confusion matrix and in the dataset 26k images around 12k images are false negatives and falso positives... Its just random but i have tried with 1 dataset 2 datasets less augmentation, dropout, sigmoid, batchnormalization and the accuracy when fitting increase but no the precision, recall, confusion matrix. I do not know what to do $\endgroup$ Oct 4, 2021 at 11:47
  • 1
    $\begingroup$ How is your accuracy 99% if half your dataframe is "false negatives" ? That's impssible : there's a confusion somewhere. Maybe the way you get accuracy is not correct, maybe you're not talking about the test accuracy, ... $\endgroup$
    – Adept
    Oct 4, 2021 at 12:01
1
$\begingroup$

As complementary information to BeamsAdept's post, you can also calculate Matthews correlation coefficient, a metric that is robust to class imbalance. It provides a single value (balanced measure), ranging between +1 and -1. In your case, scikit-learn provides an api for calculating MCC:

from sklearn.metrics import matthews_corrcoef
mcc = matthews_corrcoef(valid_labels, y_pred)

If you really have to use accuracy, you can:

  • Remove some negatives images to have approximately the same number of positive and negative images. This, of course, translates to less training data.
  • Augment new images from existing data, focusing on creating more images with a positive class. You need to careful with what methods you are going to use for your data augmentation, since you can end up with model that identifies a negative image as positive, only due to these augmentation techniques.

Forgot to mention that AUC may provide misleading insights on your model's performance when your dataset is imbalanced, since, for instance, a high number of false positives can give high AUC scores.

$\endgroup$
6
  • $\begingroup$ I dont need to use accuracy it is just for me to see if it is good trained or not. I calculated Matthews :0.005129111236164701. I already did augmentation with rotations in prior importing the dataset. $\endgroup$ Oct 4, 2021 at 11:43
  • $\begingroup$ I'm not sure about the last paragraph: a high number of false positives would give small specificity / large FPR, which puts points to the right side of the ROC space; if that happens for many cutoffs, you'll have small AUC. $\endgroup$
    – Ben Reiniger
    Oct 4, 2021 at 15:01
  • $\begingroup$ You can have a model with a high TPR (how good the model is at predicting the positive class when the actual outcome is positive) and high FPR (how likely is a positive class to be predicted when the true label is negative). In that case, AUC will be large. $\endgroup$ Oct 5, 2021 at 9:11
  • $\begingroup$ Judging from all the metrics you have computed, your model (@JavierDecenaCastillo) is random at this point. I can't spot any mistakes in your model, so the only thing that comes in mind is tweaking the hyperparameters. If I have to make a wild guess, I would say your model requires more training time, so play around with your initial LR, learning scheduler and number of epochs. Let me know of your progress. $\endgroup$ Oct 5, 2021 at 9:16
  • $\begingroup$ Ive tried with 30epochs, 128 batch, 64 batch, less LR more LR. 40min of training and the same precision, AUC, and that.. i dont know what to do $\endgroup$ Oct 5, 2021 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.