Overview
I understand the surface of the mathematics* of simple neural networks. I went through single-label image classification problems (ie using MNIST & fashion-MNIST datasets) using the native tensorflow, performed multi-label image classification using Vertex AI's AutoML, and I am aware of regression tutorials on tabular data (ie this). I can perform classification on tabular data using the same principles as used for image data.
In this question, I am asking about how to perform regression on image data.
| Tabular Data | Image Data | |
|---|---|---|
| Regression | Basic regression: Predict fuel efficiency | Relatively little literature, the theme of this question |
| Classification | Very similar NN architecture works as for image classification | See MNIST digit classification for example |
Earlier threads (ie this, this, this, this, this or this) related to this topic are either unanswered or don't provide a fully reproducible setup. This question aims to do so: in the following section, I present how I create mock data, how I train my model, and what the problem is with it.
A regression problem
What the model should estimate
I can create a large number of images with a tilted elongated rectangle on them, with some other points on the image as noise:
Given an image, I am trying to build a Tensorflow model that estimates the slope of this rectangle.
Reproducible data generation
# Imports for this and the following sections:
import PIL
import glob
import numpy as np
import tensorflow as tf
from tqdm import tqdm
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle
A simple function that creates this_many pngs as above, each 10 by 10 inch, saved with 20 dpi (so the result will be 200 by 200 pixels):
def generate_data(this_many,lim = 10,prefix=''):
for i in tqdm(range(this_many)):
# create plot with limits
plt.figure(figsize=(5,5))
plt.xlim([-lim,lim])
plt.ylim([-lim,lim])
# add tilted rectangle
angle = np.random.uniform(low=0, high=180)
plt.gca().add_patch(Rectangle((0,0),lim-1,1,angle=angle,facecolor='k'))
plt.gca().add_patch(Rectangle((0,0),-lim+1,1,angle=angle,facecolor='k'))
# add scatter plot as noise
xs = np.random.uniform(low=-lim, high=lim, size=50)
ys = np.random.uniform(low=-lim, high=lim, size=50)
plt.scatter(xs, ys, s=100, c='k')
# tidy up
plt.gca().set_aspect('equal')
plt.gca().get_yaxis().set_visible(False)
plt.gca().get_xaxis().set_visible(False)
plt.savefig(f'{prefix}sample{i:04}_angle_{int(angle*100):05}.png',dpi=20)
plt.close()
Each filename will also contain the angle at which the rectangle is rotated on the image.
#Generate 10000 such pngs:
!mkdir pngs
generate_data(10000,prefix='pngs/')
Read in image data
#List all `png`s we have generated:
pngs = glob.glob('pngs/sample*png')
Create numpy arrays from it (as here):
ims = {}
for png in pngs:
ims[png]=np.array(PIL.Image.open(png))
Let's call the arrays created from pngs questions, as the neural net will be questioned on the slope of rectangles appearing on these:
#Let's call the arrays created from pngs `questions`
questions = np.array([each for each in ims.values()]).astype(np.float32)
#Check the first color channel of the first image:
plt.imshow(questions[0][:,:,0])
It could be improved, but decent enough.
#Read in the slopes to an array:
solutions = np.array([float(each.split('_')[-1].split('.')[0])/100 for each in ims]).astype(np.float32)
#Check the slope on the image above:
solutions[0] # outputs: 100.88
Seems correct.
Where Tensorflow comes in
#Define our model:
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(4, 4, activation='sigmoid'),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(units=100/4*100/4, activation='sigmoid'),
tf.keras.layers.Dense(units=1000, activation='sigmoid'),
tf.keras.layers.Dense(units=50, activation='sigmoid'),
tf.keras.layers.Dense(units=1)
])
#Compile:
model.compile(loss='mean_squared_error',
optimizer=tf.keras.optimizers.SGD(0.01))
#Fit:
history = model.fit(questions, solutions, epochs=10, batch_size = 200, verbose=1)
#Output:
Epoch 1/10
50/50 [==============================] - 22s 442ms/step - loss: 2698.5503
Epoch 2/10
50/50 [==============================] - 21s 426ms/step - loss: 2699.3318
Epoch 3/10
50/50 [==============================] - 22s 436ms/step - loss: 2699.2109
Epoch 4/10
50/50 [==============================] - 22s 445ms/step - loss: 2701.0398
Epoch 5/10
50/50 [==============================] - 21s 422ms/step - loss: 2700.9006
Epoch 6/10
50/50 [==============================] - 22s 437ms/step - loss: 2701.7229
Epoch 7/10
50/50 [==============================] - 22s 439ms/step - loss: 2698.1746
Epoch 8/10
50/50 [==============================] - 22s 431ms/step - loss: 2695.8655
Epoch 9/10
50/50 [==============================] - 22s 444ms/step - loss: 2700.7979
Epoch 10/10
50/50 [==============================] - 22s 435ms/step - loss: 2697.8386
The fact that loss is not decreasing while being thousands high is alarming.
Test model
#Generate test data:
generate_data(30,prefix='pngs/test_')
#Read in pngs:
test_pngs = glob.glob('pngs/test_*png')
test_ims = {}
for png in test_pngs:
test_ims[png]=np.array(PIL.Image.open(png))
#Prepare test questions and solutions as before:
test_questions = np.array([each for each in test_ims.values()]).astype(np.float32)
test_solutions = np.array([float(each.split('_')[-1].split('.')[0])/100
for each in test_ims]).astype(np.float32)
#Apply model:
test_answers = model.predict(test_questions)
test_answers is:
print(test_answers)
#array([[90.65718 ],
# [90.65722 ],
# [90.65722 ],
# [90.65722 ],
# .
# .
# .
# [90.65722 ],
# [90.657196],
# [90.65721 ],
# [90.65723 ]], dtype=float32)
i.e., all of them are almost the same. Correct estimates would've been close to test_solutions:
print(test_solutions)
#array([ 21.56, 128.17, 126.59, 104.89, ... 168.03, 68.59, 124.97, 155.32], dtype=float32)
i.e. the model is completely wrong. It does not seem to be the case that simply tweaking epoch numbers or batch size is going to help. (I did try some other numbers though, but it indeed did not help.)
Question
What tensorflow architecture would allow a model to be capable of estimating the slope of these above rectangles after training?
In other words: what's wrong with the approach above?
