# Contrastive loss problem in a character-level, siamese NN model

Summary: my NN with contrastive loss does not work, need help debugging

Background: I am trying to replicate this paper

At first, I used binary crossentropy for loss, and the results were very good, but there was a caveat. The model correctly predicted which job titles were similar, but only if it had seen both titles in the training data. If it saw a given title for the first time, it would ALWAYS predict 1 (i.e. that they are similar). This is very undesirable for me product-wise.

Then, I tried the following implementation of the contrastive loss:

def contrastive_loss(y_true, y_pred):
"""Contrastive loss from Hadsell-et-al.'06
http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf
"""
margin = 1
sqaure_pred = K.square(y_pred)
margin_square = K.square(K.maximum(margin - y_pred, 0))
return K.mean(y_true * sqaure_pred + (1 - y_true) * margin_square)


However, what happens now is that the model always predicts all zeros. I run this code with print_tensor...

    sqaure_pred =  K.print_tensor(K.square(y_pred))
margin_square =  K.print_tensor(K.square(K.maximum(margin - y_pred, 0)))
return  K.print_tensor(K.mean(y_true * sqaure_pred + (1 - y_true) * margin_square))


...and this is what I saw

Any ideas what I should try next?

This is the model:

def build_blstm_encoder(params):
lstm = params['lstm']
nb_tokens = params['nb_tokens']
maxlen = params['max_seq_len']
offer_rep_dim = params['offer_rep_dim']
emb_len = params['emb_len']

input_1 = Input(shape=(maxlen,), dtype='int32')
input_2 = Input(shape=(maxlen,), dtype='int32')
emb_layer = Embedding(nb_tokens, output_dim=emb_len, input_length=maxlen, mask_zero=False)
blstm_layer = Bidirectional(LSTM(output_dim=lstm, return_sequences=True), merge_mode='concat', weights=None)
dense = Dense(offer_rep_dim, activation='relu')

blstm_encoders = []
for char_array in [input_1, input_2]:
embs = emb_layer(char_array)
blstm = blstm_layer(embs)
dropout = Dropout(0.15)(blstm)
dense_ = dense(dropout)
flatten = Flatten()(dense_)
blstm_encoders.append(flatten)

distance = Dot([1, 1], normalize=True)(blstm_encoders)
return Model([input_1, input_2], [distance])


The optimizer is just Adam() (default parameters)

Questions: Any idea what went wrong? What would you try next?