0
$\begingroup$

I'm trying to make a neural network learn a 2D shape. I have an $n\times m$ grid of points with a binary function defined on it, equal to $1$ when the point is in the shape and $0$ when it isn't. I want my network to accept as input two numbers, $x$ and $y$, and tell me with a single binary output if the point $(x, y)$ is in the shape or not.

Ultimately I'd like to have a largeish image and a somewhat complicated shape (say an disc with a hole in it) and train the network on a random collection of points, but for now I'm working with a small image, an axis-aligned rectangle, and I'm training the network on every pixel in the image.

My problem is that my network doesn't work at all. I got the Keras tutorial working and it reaches accuracy of over 90% within ten epochs, however with my code the training looks like this:

Epoch 1/10
17/17 [==============================] - 0s 932us/step - loss: 14.4782 - accuracy: 0.0506
Epoch 2/10
17/17 [==============================] - 0s 757us/step - loss: 14.4782 - accuracy: 0.0506
Epoch 3/10
17/17 [==============================] - 0s 722us/step - loss: 14.4782 - accuracy: 0.0506
Epoch 4/10
17/17 [==============================] - 0s 696us/step - loss: 14.4782 - accuracy: 0.0506
Epoch 5/10
17/17 [==============================] - 0s 653us/step - loss: 14.4782 - accuracy: 0.0506
Epoch 6/10
17/17 [==============================] - 0s 655us/step - loss: 14.4782 - accuracy: 0.0506
Epoch 7/10
17/17 [==============================] - 0s 656us/step - loss: 14.4782 - accuracy: 0.0506
Epoch 8/10
17/17 [==============================] - 0s 677us/step - loss: 14.4782 - accuracy: 0.0506
Epoch 9/10
17/17 [==============================] - 0s 751us/step - loss: 14.4782 - accuracy: 0.0506
Epoch 10/10
17/17 [==============================] - 0s 777us/step - loss: 14.4782 - accuracy: 0.0506
50/50 - 0s - loss: 14.4773 - accuracy: 0.0506

Not only does it not converge, the accuracy doesn't move at all. I tried 100 epochs and had the same results.

My code:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import Flatten, Dense
import numpy as np
import matplotlib.pyplot as pyplot

def get_coordinate_map(width, height):
    ones = np.ones((height, width))
    offset = np.array([(width / 2) * ones, (height / 2) * ones])
    coordinates = np.indices((height, width)) - offset
    return coordinates.reshape(2, width * height).T

def get_rectangle(width, height, rectangle_width, rectangle_height):
    ones = np.ones((height, width))
    offset = np.array([(width / 2) * ones, (height / 2) * ones])
    coordinates = np.indices((height, width)) - offset
    flat_coordinates = coordinates.reshape(2, width * height)
    rectangle = []
    x1, x2 = -rectangle_width / 2, rectangle_width / 2
    y1, y2 = -rectangle_height / 2, rectangle_height / 2
    for x, y in flat_coordinates.T:
        if x1 < x < x2 and y1 < y < y2:
            rectangle.append(1)
        else:
            rectangle.append(0)
    return np.array(rectangle).T

if __name__ == "__main__":
    SCALE = 40
    training_points = get_coordinate_map(SCALE, SCALE)
    training_labels = get_rectangle(SCALE, SCALE, 10, 10)

    model = keras.Sequential(
        [
            Dense(20, activation="relu"),
            Dense(1, activation="softmax")
        ]
    )
    model.compile(
        optimizer="adam",
        loss=tf.keras.losses.BinaryCrossentropy(),
        metrics=["accuracy"]
    )
    model.build((1, 2))
    model.fit(training_points[::3], training_labels[::3], epochs=10)
    
    loss, accuracy = model.evaluate(training_points, training_labels, verbose=2)

And since my get_***() functions are probably difficult to read, here's picture of the image generated by get_rectangle():

enter image description here

I'm doing this for the purposes of learning about neural networks, so I'm not interested in solutions involving using a different kind of ML technique altogether. My question is why I'm getting such strikingly unhelpful behavior and if this kind of binary classification of low dimensional data is even possible with a neural net.

$\endgroup$

1 Answer 1

1
$\begingroup$

if you use "softmax" you would actually need two neurons as the output layer, one for each label 0 or 1, complementing each other. It basically produces a probability distribution for the lables (in this case 2 labels 0,1). This link might be helpful regarding softmax. Then the target you provide should be also two columns:

    if __name__ == "__main__":
        SCALE = 40
        training_points = get_coordinate_map(SCALE, SCALE)
        training_labels = get_rectangle(SCALE, SCALE, 10, 10)

        model = keras.Sequential(
            [
                Dense(20, activation="relu"),
                Dense(2, activation="softmax")
            ]
        )
        model.compile(
            optimizer="adam",
            loss=tf.keras.losses.BinaryCrossentropy(),
            metrics=["accuracy"]
        )
        model.build((None, 2))
        model.fit(training_points[::3], 
                np.array([training_labels[::3], 1-training_labels[::3]]).T, 
                epochs=10, batch_size=20)
    
        loss, accuracy = model.evaluate(training_points, 
                    np.array([training_labels, 1-training_labels]).T, verbose=2)
        print(loss, accuracy)

result will be:

Epoch 1/10
534/534 [==============================] - 1s 2ms/step - loss: 0.3898 - acc: 0.8109
Epoch 2/10
534/534 [==============================] - 0s 88us/step - loss: 0.1154 - acc: 0.9494
Epoch 3/10
534/534 [==============================] - 0s 94us/step - loss: 0.1001 - acc: 0.9494
Epoch 4/10
534/534 [==============================] - 0s 90us/step - loss: 0.0967 - acc: 0.9494
Epoch 5/10
534/534 [==============================] - 0s 91us/step - loss: 0.0944 - acc: 0.9494
Epoch 6/10
534/534 [==============================] - 0s 82us/step - loss: 0.0922 - acc: 0.9494
Epoch 7/10
534/534 [==============================] - 0s 103us/step - loss: 0.0905 - acc: 0.9494
Epoch 8/10
534/534 [==============================] - 0s 87us/step - loss: 0.0889 - acc: 0.9494
Epoch 9/10
534/534 [==============================] - 0s 86us/step - loss: 0.0876 - acc: 0.9494
Epoch 10/10
534/534 [==============================] - 0s 90us/step - loss: 0.0861 - acc: 0.9494
0.0849673491239082 0.95

instead you can use "sigmoid" as the activation function of the output layer:

    if __name__ == "__main__":
        SCALE = 40
        training_points = get_coordinate_map(SCALE, SCALE)
        training_labels = get_rectangle(SCALE, SCALE, 10, 10)

        model = keras.Sequential(
            [
                Dense(20, activation="relu"),
                Dense(1, activation="sigmoid")
            ]
        )
        model.compile(
            optimizer="adam",
            loss=tf.keras.losses.BinaryCrossentropy(),
            metrics=["accuracy"]
        )
        model.build((None, 2))
        model.fit(training_points[::3], training_labels[::3], epochs=10, batch_size=20)
    
        loss, accuracy = model.evaluate(training_points, training_labels, verbose=2)
        print(loss, accuracy)

results:

Epoch 1/10
534/534 [==============================] - 1s 2ms/step - loss: 1.6602 - acc: 0.5187
Epoch 2/10
534/534 [==============================] - 0s 85us/step - loss: 0.6545 - acc: 0.7060
Epoch 3/10
534/534 [==============================] - 0s 87us/step - loss: 0.2457 - acc: 0.9251
Epoch 4/10
534/534 [==============================] - 0s 83us/step - loss: 0.1578 - acc: 0.9494
Epoch 5/10
534/534 [==============================] - 0s 88us/step - loss: 0.1345 - acc: 0.9494
Epoch 6/10
534/534 [==============================] - 0s 87us/step - loss: 0.1224 - acc: 0.9494
Epoch 7/10
534/534 [==============================] - 0s 84us/step - loss: 0.1149 - acc: 0.9494
Epoch 8/10
534/534 [==============================] - 0s 82us/step - loss: 0.1100 - acc: 0.9494
Epoch 9/10
534/534 [==============================] - 0s 91us/step - loss: 0.1064 - acc: 0.9494
Epoch 10/10
534/534 [==============================] - 0s 108us/step - loss: 0.1040 - acc: 0.9494
0.1027598740439862 0.949375
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.