6
$\begingroup$

How do you add more importance to some samples than others (sample weights) in Keras?

I'm not looking for class_weightwhich is a fix for unbalanced datasets.

What I currently have is:

trainingWeights which is the desired importance I want to give to each sample.

epochs = 30
batchSize = 512

# Fit model with selected data
model.fit(trainingMatrix, trainingTargets,
          batch_size=batchSize, epochs=epochs, 
          sample_weight=trainingWeights)

However the training error is much lower than before, and according to Keras' documentation:

sample_weight: Optional Numpy array of weights for the training samples, used for weighting the loss function (during training only).

As I understand it, this option only calculates the loss function differently without training the model with weights (sample importance) so how do I train a Keras model with different importance (weights) for different samples.

PD. This is a similar question xgboost: give more importance to recent samples but I would like an applicable answer to Keras.

$\endgroup$
  • $\begingroup$ Does it make sense to weight the softmax-normalization (if used) rather than the loss function? $\endgroup$ – ignatius May 10 '18 at 10:36
  • $\begingroup$ Could you explain that more thoroughly as an answer, I'm also interested in hearing other ideas. Code snippets are encouraged. $\endgroup$ – wacax May 10 '18 at 16:26
3
+50
$\begingroup$

As I understand it, this option only calculates the loss function differently without training the model with weights (sample importance) so how do I train a Keras model with different importance (weights) for different samples.

when the loss function is calculated differently that means the backprop will behave differently (more emphasis to important samples).

However the training error is much lower than before

If I understand by error you mean the average loss calculated for each batch.
the cross-entropy loss function is a proxy (differentiable) to train the model, while the zero-one loss is the one to fine-tune it.

This is a similar question xgboost: give more importance to recent samples but I would like an applicable answer to Keras

Penalizing neural networks for specific examples means give them a high probability to be included in the batches. there is no correspondence with XGboost here.

side note: maybe you could try Training with a Curriculum

Gradually transforming the training task, from an easy one (maybe convex) where examples illustrate the simpler concepts, to the target one (with more difficult examples)
The basic idea is to start small, learn easier aspects of the task or easier sub-tasks, and then gradually increase the difficulty level. From the point of view of building representations, advocated here, the idea is to learn representations that capture low-level abstractions
first, and then exploit them and compose them to learn slightly higher-level abstractions necessary to explain more complex structure in the data. By choosing which examples to present and in which order to present them to the learning system, one can guide training and remarkably increase the speed at which learning can occur. learning deep architectures for ai by yoshua bengio

$\endgroup$
  • $\begingroup$ Just to double check, as you said "when the loss function is calculated differently that means the backprop will behave differently (more emphasis to important samples)." Does that mean the resulting model will be different to the model where no weights on sample_weight were used? $\endgroup$ – wacax May 10 '18 at 16:37
  • 1
    $\begingroup$ of course, just a side note: Neural network training is non-deterministic, and converges to a different function every time it is run. Training may halt at a point where the gradient becomes small, a point where early stopping ends training to prevent overfitting, or at a point where the gradient is large but it is difficult to find a downhill step due to problems such as stochasticity or poor conditioning of the Hessian. $\endgroup$ – Fadi Bakoura May 10 '18 at 17:54
7
$\begingroup$

I'm not sure I completely understand what you want but it looks like you're trying to find an implementation for sample_weight. Well, I had tried something similar. Before I go into it, I want to mention that my error actually went up - I haven't had the time to go into why that happened.

For simplicity, lets say you know the weights you need for each class and can pass it as a dictionary. In my example I have my y_train as a one hot encoded vector. I'm using that fact to reverse engineer which class each row is pointing to and adding a weight for it. You essentially need to pass an array of weights mapping to each label (so the same length as your training data) when you fit the model.

def generate_sample_weights(training_data, class_weight_dictionary): 
    sample_weights = [class_weight_dictionary[np.where(one_hot_row==1)[0][0]] for one_hot_row in training_data]
    return np.asarray(sample_weights)

And this is passed when you call fit on the model.

model.fit(x=X_train, 
    y=y_train, 
    batch_size = 64,
    validation_data=(X_val, y_val),
    shuffle=True,
    epochs=20,
    sample_weight = generate_sample_weights(y_train, class_weights_dict)
)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.