1
$\begingroup$

I am an absolute newbie in TensorFlow but have a fair understanding of ML algorithms. I have a project to model the error characteristic of time of flight cameras. As ground truth, I have acquired a set of depth images(512x424 pixels) from a stereo set up. The the range images(512x424 pixels) from the ToF camera needs to be compared with the reference depth images. In order to learn the error characteristic I am implementing a deep neural network with the reference image pixels as training input data(median pixels as features) and the difference of reference image and range camera image as training output value. There are 3 pairs of images to train and 1 pair of images to test. I have flattened the image matrices so the training input data are 3-element lists of 217088 sized arrays.

My code works without any errors but the result is ugly. Please look into the following issues:

  1. The cost reduces nicely after the first epoch but does not change much after the second epoch.
  2. The accuracy of the test phase is horrendous.
  3. The code is extremely slow. It takes almost 2 hours for a complete run. May be it has to do with the hardware. I am running it on core i3.

My code:

import tensorflow as tf
import numpy 
import cv2
import matplotlib.pyplot as plt
import glob

refDepthImgLoc = 'M:\Internship\Scan\png\scan_dist*.png'
tofDepthImgLoc = 'M:\Internship\Scan\png\kinect_distance*.png'

numImg = 4

refDepthImg = []
tofDepthImg = []
refLoc = glob.glob(scanDistImgLoc)
tofLoc = glob.glob(tofDistImgLoc)

for refImg, tofImg in zip(refLoc, tofLoc) :
    img1 = cv2.imread(refImg, 0)
    refDepthImg.append(img1)

    img2 = cv2.imread(tofImg, 0)
    tofDepthImg.append(img2)

trainData_median = []
trainLabel = []
for i in range(len(refDepthImg)):
    tempData = cv2.medianBlur(refDepthImg[i], 3)
    trainData_median.append(tempData.ravel())
    tempLabel = refDepthImg[i] - tofDepthImg[i]
    trainLabel.append(tempLabel.ravel())

n_nodes_hl1 = 100
n_nodes_hl2 = 100
n_nodes_hl3 = 100

n_input = 1;
n_output = 1;
learning_rate = 0.01

x = tf.placeholder('float')
y = tf.placeholder('float')

def neural_network_model(data):
    hidden_1_layer = {'weights':tf.Variable(tf.random_normal([n_input, n_nodes_hl1])),
                      'biases':tf.Variable(tf.random_normal([n_nodes_hl1]))}

    hidden_2_layer = {'weights':tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2])),
                      'biases':tf.Variable(tf.random_normal([n_nodes_hl2]))}

    hidden_3_layer = {'weights':tf.Variable(tf.random_normal([n_nodes_hl2, n_nodes_hl3])),
                      'biases':tf.Variable(tf.random_normal([n_nodes_hl3]))}

    l1 = tf.add(tf.matmul(data,hidden_1_layer['weights']), hidden_1_layer['biases'])
    l1 = tf.nn.relu(l1)

    l2 = tf.add(tf.matmul(l1,hidden_2_layer['weights']), hidden_2_layer['biases'])
    l2 = tf.nn.relu(l2)

    l3 = tf.add(tf.matmul(l2,hidden_3_layer['weights']), hidden_3_layer['biases'])
    l3 = tf.nn.relu(l3)

    output = tf.reduce_sum(l3)

    return output

def train_neural_network(x):
    prediction = neural_network_model(x)
    cost = tf.reduce_sum(tf.square(prediction-y))/((numImg-1)*len(trainLabel[0]))
    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

    hm_epochs = 10

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())

        for epoch in range(hm_epochs):
            tempLoss = 0

            for i in range(numImg - 1):
                 for (X, Y) in zip(trainData_median[i], trainLabel[i]):
                    _, c = sess.run([optimizer, cost], feed_dict={x: [[X]], y: [[Y]]})
                    tempLoss += c

            print('Epoch', (epoch+1), 'completed out of',hm_epochs,'loss:',tempLoss)

        print("Testing starts now")

        test = tf.abs(prediction - y)
        i = 0;
        pred = numpy.zeros(len(trainLabel[0]));
        result = numpy.zeros(len(trainLabel[0]));
        for (X, Y) in zip(trainData_median[numImg - 1], trainLabel[numImg - 1]):
            correct, pred[i] = sess.run([test, prediction], feed_dict={x: [[X]], y: [[Y]]}) 

            if (correct < 0.5):
                result[i] = 1
            i += 1

        accuracy = tf.reduce_mean(tf.cast(result, 'float'))
        print('Accuracy:', accuracy.eval())

train_neural_network(x)

The output:

Epoch 1 completed out of 10 loss: 204681865.46
Epoch 2 completed out of 10 loss: 3188.81297796
Epoch 3 completed out of 10 loss: 3183.35926716
Epoch 4 completed out of 10 loss: 3181.37895241
Epoch 5 completed out of 10 loss: 3179.95276242
Epoch 6 completed out of 10 loss: 3178.51366003
Epoch 7 completed out of 10 loss: 3177.6227609
Epoch 8 completed out of 10 loss: 3176.69995104
Epoch 9 completed out of 10 loss: 3176.85162593
Epoch 10 completed out of 10 loss: 3177.04338937
Testing starts now
Accuracy: 0.00301721

Please comment if something is inherently wrong in the code or the entire approach to the problem is incorrect. Should I try implementing it using CNNs? Please help me in making this work. Please let me know if any further information is required.

$\endgroup$
  • $\begingroup$ a.) Do you have only 3 images as your training data? That is simply too less to learn anything. Can you by any means obtain more data? b.) Tensorflow takes too much time and produces meaningless results if the array (tensor) dimension of your prediction and labels (Y) are not the same. You may want to remove the excess square brackets ([ ]) in feed_dict={x: [[X]], y: [[Y]]} $\endgroup$ – Adarsh Chavakula Jun 29 '17 at 16:19
  • $\begingroup$ @AdarshChavakula I have 3 images of resolution 512x424 which makes 651,264 data points. Considering this to be a regression task, is the data still far too less? I used 2 sets of square brackets because the tf.matmul function needs a matrix and the (X,Y) in the for loop are scalars. Even if I remove the square brackets and use '*' instead of matmul, it changes nothing. It still takes an awful amount of time. Any suggestions? $\endgroup$ – Ijjz Jun 29 '17 at 19:14
  • $\begingroup$ @Ijjz You don't have 651,264 data points. You have three data points measured on 512x424 = 217,088 variables. If you were to run OLS on your input data, you would have three observations to estimate 217,089 parameters (217,088 for the variables plus one for the intercept). Three observations won't cut it. Once you resolve your issue about not having enough observations, however, CNNs are great for images. $\endgroup$ – Dave May 24 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.