1
$\begingroup$

Summary: my NN with contrastive loss does not work, need help debugging

Background: I am trying to replicate this paper

At first, I used binary crossentropy for loss, and the results were very good, but there was a caveat. The model correctly predicted which job titles were similar, but only if it had seen both titles in the training data. If it saw a given title for the first time, it would ALWAYS predict 1 (i.e. that they are similar). This is very undesirable for me product-wise.

Then, I tried the following implementation of the contrastive loss:

def contrastive_loss(y_true, y_pred):
    """Contrastive loss from Hadsell-et-al.'06
    http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf
    """
    margin = 1
    sqaure_pred = K.square(y_pred)
    margin_square = K.square(K.maximum(margin - y_pred, 0))
    return K.mean(y_true * sqaure_pred + (1 - y_true) * margin_square)

However, what happens now is that the model always predicts all zeros. I run this code with print_tensor...

    sqaure_pred =  K.print_tensor(K.square(y_pred))
    margin_square =  K.print_tensor(K.square(K.maximum(margin - y_pred, 0)))
    return  K.print_tensor(K.mean(y_true * sqaure_pred + (1 - y_true) * margin_square))

...and this is what I saw training_contrastive_loss

Any ideas what I should try next?

This is the model:

def build_blstm_encoder(params):
    lstm = params['lstm']
    nb_tokens = params['nb_tokens']
    maxlen = params['max_seq_len']
    offer_rep_dim = params['offer_rep_dim']
    emb_len = params['emb_len']

    input_1 = Input(shape=(maxlen,), dtype='int32')
    input_2 = Input(shape=(maxlen,), dtype='int32')
    emb_layer = Embedding(nb_tokens, output_dim=emb_len, input_length=maxlen, mask_zero=False)
    blstm_layer = Bidirectional(LSTM(output_dim=lstm, return_sequences=True), merge_mode='concat', weights=None)
    dense = Dense(offer_rep_dim, activation='relu')

    blstm_encoders = []
    for char_array in [input_1, input_2]:
        embs = emb_layer(char_array)
        blstm = blstm_layer(embs)
        dropout = Dropout(0.15)(blstm)
        dense_ = dense(dropout)
        flatten = Flatten()(dense_)
        blstm_encoders.append(flatten)

    distance = Dot([1, 1], normalize=True)(blstm_encoders)
    return Model([input_1, input_2], [distance])

The optimizer is just Adam() (default parameters)

Questions: Any idea what went wrong? What would you try next?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.