Custom loss function which is included gradient in Keras

Question

I want to make a custom loss function. Concretely, I use a 2D Convolutional neural network in Keras. So far, I've made various custom loss function by adding to losses.py. However, in this case, I encountered the trouble which is explained later.

I have attempted to make a regressor for image tasks. The data shape of both input & output are (1,128,128,2), where 1 is mini-batch size, 128 is pixel number and 2 is the number of the channels.

Anyway, I want to add new loss function to this task. I want to compute the difference of pixel gradients between answer & predicted values and add to loss function. I tried to make such like below.

def continuity(y_true, y_pred):
    import tensorflow as tf
    import numpy as np
    dx = dy = 1/128
    gridSetting = (128,128)

    u_true = y_true[0,:,:,0]
    v_true = y_true[0,:,:,1]
    u_pred = y_pred[0,:,:,0]
    v_pred = y_pred[0,:,:,1]

    cont_true = np.zeros((127,127))

    for j in range(127):
        for i in range(127):
            cont_true[i,j] = (u_true[i+1,j]-u_true[i,j])/dx + (v_true[i,j+1]-v_true[i,j])/dy

    cont_pred = np.zeros((127,127))
    for j in range(127):
        for i in range(127):
            cont_pred[i,j] = (u_pred[i+1,j]-u_pred[i,j])/dx + (v_pred[i,j+1]-v_pred[i,j])/dy

    cont_true = K.variable(value=cont_true, dtype='float64')
    cont_pred = K.variable(value=cont_pred, dtype='float64')
    cont = K.mean(K.abs(cont_true - cont_pred), axis=-1)
    mse = K.mean(K.square(y_pred - y_true), axis=-1)
    cont_mse = cont+mse
    return cont_mse

Error is written as below.

Traceback (most recent call last):
File "DSC_multi-scale_2D.py", line 107, in <module>
autoencoder.compile(optimizer='adam', loss='continuity')
File "/home/----/anaconda3/envs/tensorflow-only/lib/python3.6/site-packages/keras/engine/training.py", line 332, in compile
sample_weight, mask)
File "/home/----/anaconda3/envs/tensorflow-only/lib/python3.6/site-packages/keras/engine/training_utils.py", line 403, in weighted
score_array = fn(y_true, y_pred)
File "/home/----/anaconda3/envs/tensorflow-only/lib/python3.6/site-packages/keras/losses.py", line 86, in continuity
cont_true[i,j] = (u_true[i+1,j]-u_true[i,j])/dx + (v_true[i,j+1]-v_true[i,j])/dy
ValueError: setting an array element with a sequence.

Probably, in losses.py, the data type of u_true, v_true... doesn't support numpy.

How can I debug/fix this problem?

Answer 1

I think it is necessary to perform all operations using the backend versions, allowing Keras to perform backpropagation on every step of the function. You use the common + for example.... try K.add, which should work (based on the available arithmetic operation of the tensorflow backend).

Also, have a look at a related question, where some of the mechanics around creating a custom loss function in Keras are discussed.

The two main loops in your function that compute the gradients should be candidates for vecotisation, where you could compute the differences in one operation. Try something along these lines:

diffs = K.subtract(u_true[i+1, j], u_true[i, j])

and then the quotient in a second operation:

quot = K.divide(diffs, dx)

and then of course the same for v_true and dy.

Answer 2

@n1k31t4

Thank you for your reply and I tried your suggestion. But these errors are output as below;

AttributeError: module 'keras.backend' has no attribute 'subtract'.

According to your suggested cite, tf.subtract or tf.divide are exist so I tried to rewrite as explained later and then I encountered these errors as below;

def continuity(y_true, y_pred):
    import tensorflow as tf
    import numpy as np
    dx = dy = 1/128
    gridSetting = (128,128)

    u_true = y_true[0,:,:,0]
    v_true = y_true[0,:,:,1]
    u_pred = y_pred[0,:,:,0]
    v_pred = y_pred[0,:,:,1]

    cont_true = tf.zeros((127,127))
    for j in range(127):
        for i in range(127):
            diff_true_u = tf.subtract(u_true[i+1, j], u_true[i, j])
            cont_true_u = tf.divide(diff_true_u, dx)
            diff_true_v = tf.subtract(v_true[i, j+1], v_true[i, j])
            cont_true_v = tf.divide(diff_true_v, dy)
            cont_true_sum = cont_true_u + cont_true_v
            cont_true[i,j] = cont_true_sum

Error is as below;

File "DSC_multi-scale_2D.py", line 107, in <module>
autoencoder.compile(optimizer='adam', loss='continuity')
File "/home/----/anaconda3/envs/tensorflow-only/lib/python3.6/site-packages/keras/engine/training.py", line 332, in compile
sample_weight, mask)
File "/home/----/anaconda3/envs/tensorflow-only/lib/python3.6/site-packages/keras/engine/training_utils.py", line 403, in weighted
score_array = fn(y_true, y_pred)
File "/home/----/anaconda3/envs/tensorflow-only/lib/python3.6/site-packages/keras/losses.py", line 91, in continuity
cont_true[i,j] = cont_true_sum
TypeError: 'Tensor' object does not support item assignment

I want to substitute 'cont_true_sum' to cont_true[i,j]. How can I fix?