-1
$\begingroup$

I am creating a simple feed forward to classify if the sum of two inputs as even or odd.

I cannot change the input structure (has to be two nodes), and output structure (two nodes as well, one for even, one for odd).

Here is my code which can be run out of the box if you have PyTorch installed:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import time
from datetime import datetime
import pandas_datareader.data as web
from math import floor
from random import randrange
from keras.models import Sequential
from keras.layers import Dense, LSTM, Dropout
from keras.utils import to_categorical
from keras import optimizers
from keras import regularizers

# CONFIG 
input_sz = 2
output_sz = 2
split_pct = 0.1
iterations = 10

LEARNING_RATE = 0.001

hidden_layers = [8,8,8]  # Configurable

# END CONFIG

def sigmoid(x):
    return 1 / (1 + np.exp(-x))
    
data = []
for i in range(0, 100000):
    inputNum1 = randrange(100)
    inputNum2 = randrange(100)
    inp = [inputNum1/100, inputNum2/100]
    if (inputNum1 + inputNum2 ) % 2 == 0:
        out = [1]
    else:
        out = [0]
    data.append([inp, out])

train_data = data[:int(split_pct * len(data))]
test_data = data[int(split_pct * len(data)):]


# Build the model
model = Sequential()
for i, layer_size in enumerate(hidden_layers):
    if i == 0:
        model.add(Dense(layer_size, input_dim=input_sz, activation='relu')) # , kernel_regularizer=regularizers.l2(0.01)))
    else:
        model.add(Dense(layer_size, activation='relu')) #, kernel_regularizer=regularizers.l2(0.01)))
model.add(Dense(output_sz, activation='sigmoid'))

optimizer = optimizers.RMSprop(learning_rate=LEARNING_RATE)
model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])

# Train the model
X_train = np.array([item[0] for item in train_data])
y_train = np.array([item[1] for item in train_data])

y_train = to_categorical(y_train, num_classes=output_sz)


while (True):

    model.fit(X_train, y_train, epochs=iterations, batch_size=32, verbose=1)

    # Test the model
    X_test = np.array([item[0] for item in test_data])
    y_test = np.array([item[1] for item in test_data])
    y_test = to_categorical(y_test, num_classes=output_sz)

    score = model.evaluate(X_test, y_test, verbose=0)
    print('Test loss:', score[0])
    print('Test accuracy:', score[1])

    #input("Press Enter to continue...")

Does anyone have any recommendations to make this model train properly? Thank you

Note: I've tried RMSProp, Adam, etc - same result, stuck at 50%. Also tried increasing/decreasing the learning rate and neurons/layers, and unfortunately no result either.

$\endgroup$
2
  • $\begingroup$ You appear to believe a 2-hidden-neuron ReLU-activated neural network should be able to learn this function; why? // depending on python version, (a/100 + b/100) % 2 might be different, but I doubt it's ever what you want; what values are you expecting/getting? $\endgroup$
    – Ben Reiniger
    Commented Feb 27, 2023 at 15:30
  • $\begingroup$ the numbers are between 0 and 100, dividing by 100 will make them between 0 and 1. Relu is non-linear, it is good for these modulus type of problems. Do you have any recommendations? @BenReiniger $\endgroup$ Commented Feb 27, 2023 at 23:00

1 Answer 1

0
$\begingroup$

I see that you've decided to represent your input as:

inp = [inputNum1/100, inputNum2/100]

Why divide by 100? If you're manually transforming your input data, you have a much better choice of representation that you can choose to help your model learn by essentially turning this into an XOR problem. Also another thing, you have:

if i == 0:
        model.add(Dense(layer_size, input_dim=input_sz, activation='relu')) # , kernel_regularizer=regularizers.l2(0.01)))
    else:
        model.add(Dense(layer_size, activation='relu')) #, kernel_regularizer=regularizers.l2(0.01)))

I suggest you go back and look at why you chose ReLU as your activation function here. Does it make sense in the context of trying to output a value in the range [0, 1]?

$\endgroup$
5
  • $\begingroup$ the numbers are between 0 and 100, dividing by 100 will make them between 0 and 1. Relu is non-linear, it is good for these modulus type of problems. It does make sense to have output 0 or 1. 0 = Even, 1 = Odd, we arg max that $\endgroup$ Commented Feb 27, 2023 at 23:00
  • $\begingroup$ So as for dividing by 100: this is simply a linear transformation, so it's no different than leaving them as is. My point is that if you want to manually decide on a representation of your input data that you think would make it easier for the model to learn, it would make much more sense to have inputNum % 2 since you then effectively turn this into an XOR learning problem. As for outputting 0 or 1: yes that is indeed what you want, but since ReLU outputs in the range $(0, \inf)$, it is not a good choice of activation here. A more appropriate choice of activation would be sigmoid. $\endgroup$
    – AleksJ
    Commented Feb 28, 2023 at 1:06
  • $\begingroup$ I need to solve this problem as is. Can you propose a better arch? $\endgroup$ Commented Mar 1, 2023 at 2:22
  • $\begingroup$ Unfortunately I can't think of anything off the top of my head that isn't manually changing the inputs. Maybe you can look at the list of non-linear activation layers in PyTorch to see if there's anything that might be useful for learning a representation of even/odd numbers: pytorch.org/docs/stable/… $\endgroup$
    – AleksJ
    Commented Mar 1, 2023 at 14:31
  • $\begingroup$ Relu is one. I tried tanh.. Aah such a tricky task, I don't its possible with a simple feedforward? $\endgroup$ Commented Mar 1, 2023 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.