My training set is a set of images (either 3 channel or 1 ofc i use only one type of channel). And the labels are a sequence of points in a specific order that i want to predict from the images.

I am using a model inspired by the image captioning example on the tensorflow website. This is the also the approach that this paper takes https://arxiv.org/pdf/1901.03781.pdf

class CNN_Encoder(tf.keras.Model):
    # Since you have already extracted the features and dumped it using pickle
    # This encoder passes those features through a Fully connected layer
    def __init__(self, embedding_dim):
        super(CNN_Encoder, self).__init__()
        self.fc = tf.keras.layers.Dense(embedding_dim)

    def call(self, x):
        x = self.fc(x)
        x = tf.nn.relu(x)
        return x

class RNN_Decoder(tf.keras.Model):
    def __init__(self, embedding_dim, units, output_dim):
        super(RNN_Decoder, self).__init__()
        self.units = units

        self.gru = tf.keras.layers.GRU(self.units,
        self.fc1 = tf.keras.layers.Dense(self.units)
        self.fc2 = tf.keras.layers.Dense(output_dim)

    def call(self, x, features, hidden):

        x = tf.concat((features, x), axis=-1)
        output, state = self.gru(x)
        x = self.fc1(state)
        x = self.fc2(x)
        return x

    def reset_state(self, batch_size):
        return tf.zeros((batch_size, self.units))

def train_step(img_tensor, target):
    loss = 0

    hidden = decoder.reset_state(batch_size=target.shape[0])
    dec_input = tf.expand_dims([[0., 0.]] * target.shape[0], 1)
    with tf.GradientTape() as tape:

        features = encoder(img_tensor)
        for i in (range(1, target.shape[1])):
            predictions = decoder(dec_input, features, hidden)
            loss += loss_function(target[:, i], predictions)

            # using teacher forcing
            dec_input = tf.expand_dims(target[:, i], 1)
    total_loss = (loss / int(target.shape[1]))
    trainable_variables = encoder.trainable_variables + decoder.trainable_variables
    gradients = tape.gradient(loss, trainable_variables)
    optimizer.apply_gradients(zip(gradients, trainable_variables))
    return loss, total_loss

batch_size = 8
for epoch in tqdm(range(start_epoch, EPOCHS)):
    start = time.time()
    total_loss = 0

    for (batch, (img_tensor, target)) in enumerate((data_generator(preds_t, labels_t))):
        img_tensor = img_tensor.reshape((-1, 1, 128*128))
        batch_loss, t_loss = train_step(img_tensor, target)
        total_loss += t_loss

        if batch % 100 == 0:
            print ('Epoch {} Batch {} Loss {:.4f}'.format(
              epoch + 1, batch, batch_loss.numpy() / int(target.shape[1])))
        if batch == 10000:

    # storing the epoch end loss value to plot later
    #loss_plot.append(total_loss / num_steps)

    if epoch % 5 == 0:

    print ('Epoch {} Loss {:.6f}'.format(epoch + 1,
    print ('Time taken for 1 epoch {} sec\n'.format(time.time() - start))

For the features vector. I am extracting the last layer of a unet. So each image has a size 1x128x128. I reshape it to be 1x1x128*128. Which i then pass through a fully connected layer. The shape then becomes 1x1x256

My labels i want to predict are image coordinates so (x, y). The input to the gru layer is the concatenated 1x1x256 , 1x1x2 (t-1 coordinates). Which i then further pass through a 2 layer fc layer with output dimension 2 for the 2 coordinates. I have removed attention for now to get a simpler model. I normalize my images. I pad the coordinate sequences with 0,0 for the start -1, -1 for the end and -2,-2 for the regular padding to get uniform sequence length of 350x2.

The network doesnt seem to learn much. I just get a few points scattered diagonally across the image. The biggest difference i see with the image captioning model is that the words can be converted to embeddings and then you have a 128 image features 128 word features being concatenated and fed into the lstm. In my case the sequence information is just 1 entry. Could that be the reason that the network is not learning much.

If someone has any insights into what i should change that would be great


The approach itself looks fine. The conversion of words to embeddings is just necessary step in a image captioning exercise & since we have the sequence numbers already in your use case, they are in a sense are already encoded and ready to be used (as in a time-series non NLP prediction, like stock market predictions etc). By itself that should not make a difference. You may want to re-look at your labelled data to see if it has enough information and features for the model to train itself. Also see if the loss function can be more optimized

| improve this answer | |
New contributor
Allohvk is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.