5
$\begingroup$

Overview

I understand the surface of the mathematics* of simple neural networks. I went through single-label image classification problems (ie using MNIST & fashion-MNIST datasets) using the native , performed multi-label image classification using Vertex AI's AutoML, and I am aware of regression tutorials on tabular data (ie this). I can perform classification on tabular data using the same principles as used for image data.

In this question, I am asking about how to perform regression on image data.

Tabular Data Image Data
Regression Basic regression: Predict fuel efficiency Relatively little literature, the theme of this question
Classification Very similar NN architecture works as for image classification See MNIST digit classification for example

Earlier threads (ie this, this, this, this, this or this) related to this topic are either unanswered or don't provide a fully reproducible setup. This question aims to do so: in the following section, I present how I create mock data, how I train my model, and what the problem is with it.


A regression problem

What the model should estimate

I can create a large number of images with a tilted elongated rectangle on them, with some other points on the image as noise:

enter image description here

Given an image, I am trying to build a Tensorflow model that estimates the slope of this rectangle.

Reproducible data generation

# Imports for this and the following sections:
import PIL
import glob
import numpy as np
import tensorflow as tf
from tqdm import tqdm
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle

A simple function that creates this_many pngs as above, each 10 by 10 inch, saved with 20 dpi (so the result will be 200 by 200 pixels):

def generate_data(this_many,lim = 10,prefix=''):

    for i in tqdm(range(this_many)):

        # create plot with limits
        plt.figure(figsize=(5,5))
        plt.xlim([-lim,lim])
        plt.ylim([-lim,lim])

        # add tilted rectangle
        angle = np.random.uniform(low=0, high=180)
        plt.gca().add_patch(Rectangle((0,0),lim-1,1,angle=angle,facecolor='k'))
        plt.gca().add_patch(Rectangle((0,0),-lim+1,1,angle=angle,facecolor='k'))

        # add scatter plot as noise
        xs = np.random.uniform(low=-lim, high=lim, size=50)
        ys = np.random.uniform(low=-lim, high=lim, size=50)
        plt.scatter(xs, ys, s=100, c='k')

        # tidy up
        plt.gca().set_aspect('equal')
        plt.gca().get_yaxis().set_visible(False)
        plt.gca().get_xaxis().set_visible(False)

        plt.savefig(f'{prefix}sample{i:04}_angle_{int(angle*100):05}.png',dpi=20)
        plt.close()

Each filename will also contain the angle at which the rectangle is rotated on the image.

#Generate 10000 such pngs:
!mkdir pngs
generate_data(10000,prefix='pngs/')

Read in image data

#List all `png`s we have generated:
pngs = glob.glob('pngs/sample*png')

Create arrays from it (as here):

ims = {}
for png in pngs:
    ims[png]=np.array(PIL.Image.open(png))

Let's call the arrays created from pngs questions, as the neural net will be questioned on the slope of rectangles appearing on these:

#Let's call the arrays created from pngs `questions`
questions = np.array([each for each in ims.values()]).astype(np.float32)

#Check the first color channel of the first image:
plt.imshow(questions[0][:,:,0])

enter image description here

It could be improved, but decent enough.

#Read in the slopes to an array:
solutions = np.array([float(each.split('_')[-1].split('.')[0])/100 for each in ims]).astype(np.float32)

#Check the slope on the image above:
solutions[0] # outputs: 100.88

Seems correct.

Where Tensorflow comes in

#Define our model:

model = tf.keras.Sequential([
  tf.keras.layers.Conv2D(4, 4, activation='sigmoid'),
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(units=100/4*100/4, activation='sigmoid'),
  tf.keras.layers.Dense(units=1000, activation='sigmoid'),
  tf.keras.layers.Dense(units=50, activation='sigmoid'),
  tf.keras.layers.Dense(units=1)
])

#Compile:
model.compile(loss='mean_squared_error',
              optimizer=tf.keras.optimizers.SGD(0.01))

#Fit:
history = model.fit(questions, solutions, epochs=10, batch_size = 200, verbose=1)
#Output:

Epoch 1/10
50/50 [==============================] - 22s 442ms/step - loss: 2698.5503
Epoch 2/10
50/50 [==============================] - 21s 426ms/step - loss: 2699.3318
Epoch 3/10
50/50 [==============================] - 22s 436ms/step - loss: 2699.2109
Epoch 4/10
50/50 [==============================] - 22s 445ms/step - loss: 2701.0398
Epoch 5/10
50/50 [==============================] - 21s 422ms/step - loss: 2700.9006
Epoch 6/10
50/50 [==============================] - 22s 437ms/step - loss: 2701.7229
Epoch 7/10
50/50 [==============================] - 22s 439ms/step - loss: 2698.1746
Epoch 8/10
50/50 [==============================] - 22s 431ms/step - loss: 2695.8655
Epoch 9/10
50/50 [==============================] - 22s 444ms/step - loss: 2700.7979
Epoch 10/10
50/50 [==============================] - 22s 435ms/step - loss: 2697.8386

The fact that loss is not decreasing while being thousands high is alarming.

Test model

#Generate test data:
generate_data(30,prefix='pngs/test_')

#Read in pngs:
test_pngs = glob.glob('pngs/test_*png')

test_ims = {}
for png in test_pngs:
    test_ims[png]=np.array(PIL.Image.open(png))

#Prepare test questions and solutions as before:
test_questions = np.array([each for each in test_ims.values()]).astype(np.float32)
test_solutions = np.array([float(each.split('_')[-1].split('.')[0])/100
                            for each in test_ims]).astype(np.float32)

#Apply model:
test_answers = model.predict(test_questions)

test_answers is:

print(test_answers)
#array([[90.65718 ],
#        [90.65722 ],
#        [90.65722 ],
#        [90.65722 ],
#        .
#        .
#        .
#        [90.65722 ],
#        [90.657196],
#        [90.65721 ],
#        [90.65723 ]], dtype=float32)

i.e., all of them are almost the same. Correct estimates would've been close to test_solutions:

print(test_solutions)
#array([ 21.56, 128.17, 126.59, 104.89,  ... 168.03,  68.59, 124.97, 155.32], dtype=float32)

i.e. the model is completely wrong. It does not seem to be the case that simply tweaking epoch numbers or batch size is going to help. (I did try some other numbers though, but it indeed did not help.)


Question

What tensorflow architecture would allow a model to be capable of estimating the slope of these above rectangles after training?

In other words: what's wrong with the approach above?


* Excellent backpropagation explanations here and here.

$\endgroup$
1
  • $\begingroup$ Let me know if there is an issue with reproducibility. $\endgroup$
    – zabop
    Jan 1, 2022 at 0:41

1 Answer 1

6
$\begingroup$

I think the issue is mostly with your network architecture.

You are using only one convolutional layer, and you are using all sigmoid activations!

Adding more convolutional layers, changing the activations from sigmoid to relu, and changing the optimizer to Adam gives me a loss below 5 after 30 epochs:

model = tf.keras.Sequential([
  tf.keras.layers.Conv2D(3, 3, activation='relu'),
  tf.keras.layers.Conv2D(3, 3, activation='relu'),
  tf.keras.layers.MaxPooling2D(2),
  tf.keras.layers.Conv2D(3, 3, activation='relu'),
  tf.keras.layers.Conv2D(3, 3, activation='relu'),
  tf.keras.layers.MaxPooling2D(2),
  tf.keras.layers.Conv2D(3, 3, activation='relu'),
  tf.keras.layers.Conv2D(3, 3, activation='relu'),
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(units=512, activation='relu'),
  tf.keras.layers.Dense(units=256, activation='relu'),
  tf.keras.layers.Dense(units=64, activation='relu'),
  tf.keras.layers.Dense(units=1)
])

model.compile(loss='mean_squared_error', optimizer="adam")

history = model.fit(questions, solutions, epochs=30, batch_size=200, verbose=1)

Which gives the following training output for the last 5 epochs:

Epoch 25/30
50/50 [==============================] - 3s 54ms/step - loss: 4.8650
Epoch 26/30
50/50 [==============================] - 3s 57ms/step - loss: 5.5044
Epoch 27/30
50/50 [==============================] - 3s 56ms/step - loss: 6.0381
Epoch 28/30
50/50 [==============================] - 3s 55ms/step - loss: 4.7235
Epoch 29/30
50/50 [==============================] - 3s 54ms/step - loss: 4.4355
Epoch 30/30
50/50 [==============================] - 3s 55ms/step - loss: 4.1494
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.