1
$\begingroup$

I want to create an AI to classify images with a large set of labels (1000+ labels). However, the labels in the data set are correct but each image is not fully labelled. This means that each image's labels are correct but incomplete.

For example, an image may only have labels for classes A, B, and C, but would also fit labels D and E. This means that images may fit labels despite not being correctly assigned them.

I need an AI/training method which accounts for these inaccuracies by giving more weight to present labels and less weight to labels not present.

Is there any way to do this?

$\endgroup$
1

1 Answer 1

-1
$\begingroup$

I need an AI/training method which accounts for these inaccuracies by giving more weight to present labels and less weight to labels not present.

This is a multi-label classification problem. One thing you can try is to treat the problem as $N$ simultaneous binary classification problems, where $N$ is the number of classes. Then you can define a custom loss function that weights positive examples higher than negative examples.

Here's a concrete example in keras. First, we'll define a neural net that takes an image as input and has $N$ sigmoid outputs, one for each class.

from tensorflow.keras import Sequential
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import MaxPooling2D

labels = ["a", "b", "c", . . .]

INPUT_SHAPE = (256, 256, 1)
NUM_CLASSES = len(labels)

model = models.Sequential()

model.add(
    layers.Conv2D(32,(5,5),activation=’relu’, input_shape=INPUT_SHAPE)
)
model.add(layers.MaxPooling2D((2, 2)))

model.add(layers.Conv2D(64, (5, 5), activation=’relu’))
model.add(layers.MaxPooling2D((2, 2)))

model.add(layers.Flatten())
model.add(layers.Dense(NUM_CLASSES, activation=’sigmoid’))

Again, I want to emphasize that the final layer has sigmoid activation, not softmax like you would see in multiclass classification. This means that the model will output a probability between 0 and 1 for each class.

Next up we need a custom loss function that weighs positive labels higher than negative/missing labels. For normal multi-label problems, binary cross-entropy loss is commonly used. It is defined as:

$$L = -\sum_{c=1}^{N}[y_c log(p_c) + (1 - y_c) log(1 - p_c)]$$

where

  • $N$ is the number of classes
  • $y_c$ is binary indicator (0 or 1) if class label $c$ is the correct classification for the example
  • $p_c$ is the predicted probability that the example belongs to class $c$ (i.e. the model output for class $c$)

It is easy to modify this loss function to weigh positive and negative examples differently. Just add coefficients to the positive component of the loss and the negative component of the loss:

$$L = -\sum_{c=1}^{N}[w_p * y_c log(p_c) + w_n * (1 - y_c) log(1 - p_c)]$$

where

  • $w_p$ is the weight for positive examples
  • $w_n$ is the weight for negative examples

Technically, $w_p$ and $w_n$ are hyperparameters that should be tuned during training. But for our purposes, let's say $w_p = 1$ and $w_n = 0.5$. Now we are ready to define the custom loss function:

from tensorflow.keras import backend as K

w_p = 1.0
w_n = 0.5

def custom_loss(y_true, y_pred):
    '''  Weighted multi-label cross-entropy

    Args:
        y_true: true labels, one-hot-encoded
        y_pred: labels predicted by the model 
    '''
    loss = float(0)
    
    for i, label in enumerate(labels):
        positive_term = w_p * y_true[i] * K.log(y_pred[i] + K.epsilon())
        negative_term = w_n * (1 - y_true[i]) * K.log(1 - y_pred[i] + K.epsilon())
        loss -= (positive_term + negative_term)
    return loss

The last step is to compile and train the model with this loss function:

from tf.keras.optimizers import Adam
from tensorflow.keras.preprocessing.image import ImageDataGenerator

N_EPOCHS = 100

my_data_generator = ImageDataGenerator(rescale=1./255)
target_size = INPUT_SHAPE[0:2]

train_generator = train_datagen.flow_from_directory(
    'data/train/',
    target_size=target_size,
    batch_size=32
)

model.compile(
    optimizer=Adam(),
    loss=custom_loss
)

history = model.fit(
    my_data_generator,
    epochs=N_EPOCHS,
)

Ta da! A convolutional network with a custom loss function that gives more weight to present labels. Obviously you can re-configure the network architecure and the positive/negative weights to better fit your use-case

$\endgroup$
4
  • 1
    $\begingroup$ Thanks for this. The only problem I have is that the custom loss function you supplied throws up an error. "OperatorNotAllowedInGraphError: iterating over tf.Tensor is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature." in the line: "for i, label in enumerate(labels):" (I would upvote the answer but I'm too new to the site) $\endgroup$
    – Liam
    Jun 15, 2022 at 2:04
  • $\begingroup$ Thanks @Liam, glad it was helpful! In my sample code, labels was a vanilla python list. Is your labels a tensor? You can probably fix that error by converting labels to a numpy array on that line. Something like for i, label in enumerate(labels.numpy()) $\endgroup$
    – zachdj
    Jun 15, 2022 at 19:50
  • $\begingroup$ I wrote the answer without actually running any of the code, so please let me know if it works! I'll feel vindicated if my "-1" upvoted answer works well :) $\endgroup$
    – zachdj
    Jun 15, 2022 at 19:51
  • 1
    $\begingroup$ I rewrote the function as loss = -(w_p * y_true * K.log(y_pred + K.epsilon())) + (w_n * (1 - y_true) * K.log(1 - y_pred + K.epsilon())) to remove the for loop. This runs but I don't know if this is correct. $\endgroup$
    – Liam
    Jun 16, 2022 at 3:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.