# Can my neural network learn conditional rules when classifying?

I'm concerned that I'm attempting the impossible with my neural network. This is the scenario:

I have a 2D square world. In it, I create five circles of random size and position. I then classify one of them as the correct answer, based on the following rules:

• If any circle's radius is > THRESHOLD, I choose the largest circle
• Otherwise, I choose the circle with the origin nearest the center

I send the inputs as serial coordinates, like this: [X0, Y0, RADIUS0, X1, Y1, RADIUS1, ...].

The output is a one-hot array, e.g. [0, 0, 1, 0, 0].

I've modeled this in TensorFlow without success. My best scoring result appears to always choose the largest circle, ignoring the else clause of the arbitrary rule.

Am I fundamentally misunderstanding the capabilities of neural networks? I've tried many (many) different configurations (layer counts, node counts, activation functions ... you name it). All of my networks have been feed-forward, so far.

Thanks in advance for any insight!

Here are some details of my network and data:

• I have tried with up to 500k cases. I separate 10% for generalization checks after training, and train on the remaining 90% with a 50/50 validation split.
• I've tried with the test data weighted 75% toward rule A, 50/50, and 75% toward ruleB.
• I've tried 0-10 hidden layers, and neuron counts from 2 to 256 (each hidden layer gets the same number of neurons).
• I change the number epochs as time allows, but generally it's 10-100. My longest runs have been several hours (with giant case numbers, and dropouts to prevent overfitting).
• I've used batch sizes of 1-50.
• I've tried learning rates of 0.0001 - 0.1.
• I'm currently using ReLU activation, initializing bias to const(0.1) and kernel w/ heNormal. I have tried several other approaches for all three.
• I standardize the inputs to center on zero w/ variance of one.
• The loss function is categoricalCrossentropy.
• This information lacks a lot of key pieces of information: how large is your training dataset? What percentage of the training samples are representative of the second rule? What is the architecture of your neural network? And the hyperparameters?
– noe
Oct 18 '20 at 16:24
• Sorry - detailed information added. Oct 18 '20 at 16:51
• Is there any reason why you are trying to use a neural net to do this? It would seem to me that some feature engineering and a simple if...else statement would suffice. Mar 20 at 11:23
• Yes, it's specifically an exercise to learn about neural networks. Or I should say it was. I solved this problem (it was a mistake in an activation function). I'll add an answer to close this out. Mar 21 at 18:06

Centering the data on zero and scaling to variance of one helps with a lot of classification problems but in this case it would remove information that's needed to solve your problem as I understand it.

Another possible problem is the loss function, which you don't mention at all in your question. I would suggest something that stays fairly high when your neural network is only learning one of the two rules.

• Sorry again - my loss function is 'categoricalCrossentropy', and my optimizer is Adam; should have included those details. Oct 19 '20 at 2:32
• I will try without the standardization, and let you know. Thanks for the idea! Do you have any alternatives for "something that stays fairly high"? I'm not sure what that means, exactly, but I'm very eager to learn. Oct 19 '20 at 2:35
• What I was thinking was you could do something like calculate two separate categoricalCrossEntropy losses for each batch (one for small circles and one for circles above the threshold), multiply them together, and negate. So the final loss would be -(categoricalCrossEntropy(rule1) * categoricalCrossEntropy(rule2)) Oct 19 '20 at 2:42
• I just realised that cross entropy loss is always positive not negative, so you would just multiply them rather than negate. Oct 19 '20 at 2:50
• I've done a few runs without standardization, and it hasn't solved the problem unfortunately. The results are more accurate, however. Thanks! Oct 19 '20 at 21:15

Alrighty, I wrote some code to emulate your problem. I found the same issues, and so simplified the problem. When I modified the label function to instead always choose the biggest radius regardless of the arbitrary rule, I found that it still could not figure it out, and instead would converge to predicting 0.2 for each of the 5 circles. It appears that if you don't order the circles at the input, then the network cannot differentiate between them. This makes sense if you think about the flow through the densely connected network. There may be some success if we try to order the circles before inputting them.

import numpy as np
from tqdm import tqdm

N_CIRCLES = 5
CENTRE_RANGE = 1
RMIN, RMAX = 0.1, 0.5
THRESHOLD = 0.45

def label(x):
# If above threshold, then choose largest circle
if np.any(x[:5] > THRESHOLD):
return np.argmax(x[:5])

# Else, choose the circle nearest to (0, 0)
return np.argmax([np.linalg.norm(x[i:i+2]) for i in range(N_CIRCLES, 3*N_CIRCLES, 2)])

def generate_sample():
# {r0, r1, r2, r3, r4, x0, y0, x1, y1, x2, y2, x3, y3, x4, y4}
x = np.concatenate((np.random.uniform(RMIN, RMAX, N_CIRCLES),
np.random.uniform(-CENTRE_RANGE, CENTRE_RANGE, 2*N_CIRCLES)))

return x, label(x)

def generate_samples(n):
x = np.zeros((n, N_CIRCLES*3))
y = np.zeros(n)

for i in range(n):
x[i], y[i] = generate_sample()

return x, y

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
def __init__(self):
super().__init__()
# Kernel size 5
self.fc1 = nn.Linear(3*N_CIRCLES, 32)
self.fc2 = nn.Linear(32, 64)
self.fc3 = nn.Linear(64, N_CIRCLES)

def forward(self, x):
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = F.relu(self.fc3(x))
return F.softmax(x, dim=1)

net = Net()

import torch.optim as optim

loss_function = nn.MSELoss()

BATCH_SIZE = 100
EPOCHS = 1_000

losses = []
for epoch in tqdm(range(EPOCHS)):
X, y = generate_samples(BATCH_SIZE)
y = np.array(y, dtype=int)

ohe = np.zeros((y.size, y.max()+1))
ohe[np.arange(y.size), y] = 1

X = torch.Tensor(X).view(-1, 3*N_CIRCLES)
y = torch.Tensor(ohe)

yhat = net(X)
loss = loss_function(yhat, y)
loss.backward()
optimizer.step()

losses.append(float(loss.detach().numpy()))

import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

fig, ax = plt.subplots(figsize=(20, 10))
ax.plot(losses)
plt.show()
$$$$
`
• Wow, thanks for the code sample! Regarding ordering - how do you determine order? Do you mean to sort them by size or distance-from-center? That also feels like a form of 'cheating' to me. The network wouldn't really learn the rule, but rather learn to always choose the first (or last) sorted entry. Oct 19 '20 at 21:29
• A bit of background - on my first attempts at this I actually did sort the circles so that the correct answer was always first (as a side effect of my answer-choosing preprocessing). And the network indeed learned to always choose the first entry, almost immediately. Oct 19 '20 at 21:31

Yes, they absolutely can 'learn' conditional rules.

It turns out I had a bug in an activation function, and I (eventually) got this network to learn this problem. It learned it incredibly well, and very quickly. Amazing things, these NNs :) Big thanks to those that answered and commented!

Yes, the Universal Approximation Theorem states that a neural network can learn any function in $$R^n$$ with one hidden layer and a finite number of neurons with a non-linear activation function. There are many things that can go wrong with training a network, for one, have you tried graphing its performance over time and seeing if it is converging?

• This is wrong. Please have a look at the correct theorem. Oct 18 '20 at 12:12
• Yes, I watch the loss and accuracy values very closely. They attenuate fairly quickly (depending on the config of the run). Oct 18 '20 at 16:36
• Sure there are some nuances, but I don't think what I've written is wrong. Check out en.wikipedia.org/wiki/Universal_approximation_theorem Oct 18 '20 at 23:13
• As I understand it, that theorem pertains to continuous functions. I wonder if I am not using a continuous function? To be honest, I don't know how to visualize a function with 15 inputs and three outputs. I have no idea whether it's continuous, or whether it's solvable via NN (thus the question). Oct 19 '20 at 2:18
• You're right that it's for continuous functions, and although your problem is not continuous, the approximations of the NN should certainly be good enough for this problem. Oct 19 '20 at 2:36