0
$\begingroup$

I have been trying to replace the Iris Flower dataset with the Palmer Penguin dataset for a neural network tutorial. I am using the tutorial at https://www.kaggle.com/antmarakis/another-neural-network-from-scratch

The Palmer Penguin dataset should be a good replacement for the Iris Flower dataset because they both have 4 input variables and three species for the output. So I removed the rows with missing data from the penguin dataset and reduced it to 50 rows for each class to resemble the Iris Flower dataset. Unfortunately the training, validation, and testing accuracy are not very good and I cannot figure out how to improve it. I have tried removing the body mass in grams since it has far larger values than the other measurements in millimeters. I cannot get good predictions for data rows not used in the training.

Here is my modified code for the neural network:

import numpy as np
import pandas as pd

penguins = pd.read_csv("data/penguins_size_no_missing_extracted.csv")
penguins = penguins.sample(frac=1).reset_index(drop=True) # Shuffle

X = penguins[['culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm']]
X = np.array(X)
print(X[:5])

from sklearn.preprocessing import OneHotEncoder
one_hot_encoder = OneHotEncoder(sparse=False)

Y = penguins.species
Y = one_hot_encoder.fit_transform(np.array(Y).reshape(-1, 1))
print(Y[:5])

from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.15)
X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.1)


def NeuralNetwork(X_train, Y_train, X_val=None, Y_val=None, epochs=10, nodes=[], lr=0.15):
    hidden_layers = len(nodes) - 1
    weights = InitializeWeights(nodes)

    for epoch in range(1, epochs+1):
        weights = Train(X_train, Y_train, lr, weights)

        if(epoch % 20 == 0):
            print("Epoch {}".format(epoch))
            print("Training Accuracy: {}".format(Accuracy(X_train, Y_train, weights)))
            if X_val.any():
                print("Validation Accuracy: {}".format(Accuracy(X_val, Y_val, weights)))
                print()
            
    return weights

def InitializeWeights(nodes):
    """Initialize weights with random values in [-1, 1] (including bias)"""
    layers, weights = len(nodes), []
    
    for i in range(1, layers):
        w = [[np.random.uniform(-1, 1) for k in range(nodes[i-1] + 1)]
              for j in range(nodes[i])]
        weights.append(np.matrix(w))
    
    return weights
    
def ForwardPropagation(x, weights, layers):
    activations, layer_input = [x], x
    for j in range(layers):
        activation = Sigmoid(np.dot(layer_input, weights[j].T))
        activations.append(activation)
        layer_input = np.append(1, activation) # Augment with bias
    
    return activations    
    
def BackPropagation(y, activations, weights, layers):
    outputFinal = activations[-1]
    error = np.matrix(y - outputFinal) # Error at output
    
    for j in range(layers, 0, -1):
        currActivation = activations[j]
        
        if(j > 1):
            # Augment previous activation
            prevActivation = np.append(1, activations[j-1])
        else:
            # First hidden layer, prevActivation is input (without bias)
            prevActivation = activations[0]
        
        delta = np.multiply(error, SigmoidDerivative(currActivation))
        weights[j-1] += lr * np.multiply(delta.T, prevActivation)

        w = np.delete(weights[j-1], [0], axis=1) # Remove bias from weights
        error = np.dot(delta, w) # Calculate error for current layer
    
    return weights
    
def Train(X, Y, lr, weights):
    layers = len(weights)
    for i in range(len(X)):
        x, y = X[i], Y[i]
        x = np.matrix(np.append(1, x)) # Augment feature vector
        
        activations = ForwardPropagation(x, weights, layers)
        weights = BackPropagation(y, activations, weights, layers)

    return weights
    
def Sigmoid(x):
    return 1 / (1 + np.exp(-x))

def SigmoidDerivative(x):
    return np.multiply(x, 1-x)
    
def Predict(item, weights):
    layers = len(weights)
    item = np.append(1, item) # Augment feature vector
    
    ##_Forward Propagation_##
    activations = ForwardPropagation(item, weights, layers)
    
    outputFinal = activations[-1].A1
    index = FindMaxActivation(outputFinal)

    # Initialize prediction vector to zeros
    y = [0 for i in range(len(outputFinal))]
    y[index] = 1  # Set guessed class to 1

    return y # Return prediction vector


def FindMaxActivation(output):
    """Find max activation in output"""
    m, index = output[0], 0
    for i in range(1, len(output)):
        if(output[i] > m):
            m, index = output[i], i
    
    return index
    
def Accuracy(X, Y, weights):
    """Run set through network, find overall accuracy"""
    correct = 0

    for i in range(len(X)):
        x, y = X[i], list(Y[i])
        guess = Predict(x, weights)

        if(y == guess):
            # Guessed correctly
            correct += 1

    return correct / len(X)
    
f = len(X[0]) # Number of features
o = len(Y[0]) # Number of outputs / classes

layers = [f, 4, 8, o] # Number of nodes in layers
lr, epochs = 0.15, 100

weights = NeuralNetwork(X_train, Y_train, X_val, Y_val, epochs=epochs, nodes=layers, lr=lr);                        
print("Testing Accuracy: {}".format(Accuracy(X_test, Y_test, weights)))
print()

# Make predictions
x = [41.5,18.5,201]
guess = Predict(x, weights)
print("Prediction: ", end='')
print(guess)
x = [50.2,18.7,198]
guess = Predict(x, weights)
print("Prediction: ", end='')
print(guess)
x = [49.9,16.1,213]
guess = Predict(x, weights)
print("Prediction: ", end='')
print(guess)

The output is:

[[ 59.6  17.  230. ]
 [ 49.   19.5 210. ]
 [ 45.3  13.7 210. ]
 [ 43.2  16.6 187. ]
 [ 45.2  15.8 215. ]]
[[0. 0. 1.]
 [0. 1. 0.]
 [0. 0. 1.]
 [0. 1. 0.]
 [0. 0. 1.]]
Epoch 20
Training Accuracy: 0.3508771929824561
Validation Accuracy: 0.38461538461538464

Epoch 40
Training Accuracy: 0.3508771929824561
Validation Accuracy: 0.38461538461538464

Epoch 60
Training Accuracy: 0.3508771929824561
Validation Accuracy: 0.38461538461538464

Epoch 80
Training Accuracy: 0.32456140350877194
Validation Accuracy: 0.23076923076923078

Epoch 100
Training Accuracy: 0.32456140350877194
Validation Accuracy: 0.23076923076923078

Testing Accuracy: 0.43478260869565216

Prediction: [0, 1, 0]
Prediction: [0, 1, 0]
Prediction: [0, 1, 0]

The CSV file has the following data:

species,island,culmen_length_mm,culmen_depth_mm,flipper_length_mm,body_mass_g,sex
Adelie,Torgersen,39.5,17.4,186,3800,FEMALE
Adelie,Torgersen,40.3,18,195,3250,FEMALE
Adelie,Torgersen,36.7,19.3,193,3450,FEMALE
Adelie,Torgersen,39.3,20.6,190,3650,MALE
Adelie,Torgersen,38.9,17.8,181,3625,FEMALE
Adelie,Torgersen,39.2,19.6,195,4675,MALE
Adelie,Torgersen,41.1,17.6,182,3200,FEMALE
Adelie,Torgersen,38.6,21.2,191,3800,MALE
Adelie,Torgersen,34.6,21.1,198,4400,MALE
Adelie,Torgersen,36.6,17.8,185,3700,FEMALE
Adelie,Torgersen,38.7,19,195,3450,FEMALE
Adelie,Torgersen,42.5,20.7,197,4500,MALE
Adelie,Torgersen,34.4,18.4,184,3325,FEMALE
Adelie,Torgersen,46,21.5,194,4200,MALE
Adelie,Biscoe,37.8,18.3,174,3400,FEMALE
Adelie,Biscoe,37.7,18.7,180,3600,MALE
Adelie,Biscoe,35.9,19.2,189,3800,FEMALE
Adelie,Biscoe,38.2,18.1,185,3950,MALE
Adelie,Biscoe,38.8,17.2,180,3800,MALE
Adelie,Biscoe,35.3,18.9,187,3800,FEMALE
Adelie,Biscoe,40.6,18.6,183,3550,MALE
Adelie,Biscoe,40.5,17.9,187,3200,FEMALE
Adelie,Biscoe,37.9,18.6,172,3150,FEMALE
Adelie,Biscoe,40.5,18.9,180,3950,MALE
Adelie,Dream,39.5,16.7,178,3250,FEMALE
Adelie,Dream,37.2,18.1,178,3900,MALE
Adelie,Dream,39.5,17.8,188,3300,FEMALE
Adelie,Dream,40.9,18.9,184,3900,MALE
Adelie,Dream,36.4,17,195,3325,FEMALE
Adelie,Dream,39.2,21.1,196,4150,MALE
Adelie,Dream,38.8,20,190,3950,MALE
Adelie,Dream,42.2,18.5,180,3550,FEMALE
Adelie,Dream,37.6,19.3,181,3300,FEMALE
Adelie,Dream,39.8,19.1,184,4650,MALE
Adelie,Dream,36.5,18,182,3150,FEMALE
Adelie,Dream,40.8,18.4,195,3900,MALE
Adelie,Dream,36,18.5,186,3100,FEMALE
Adelie,Dream,44.1,19.7,196,4400,MALE
Adelie,Dream,37,16.9,185,3000,FEMALE
Adelie,Dream,39.6,18.8,190,4600,MALE
Adelie,Dream,41.1,19,182,3425,MALE
Adelie,Dream,36,17.9,190,3450,FEMALE
Adelie,Dream,42.3,21.2,191,4150,MALE
Adelie,Biscoe,39.6,17.7,186,3500,FEMALE
Adelie,Biscoe,40.1,18.9,188,4300,MALE
Adelie,Biscoe,35,17.9,190,3450,FEMALE
Adelie,Biscoe,42,19.5,200,4050,MALE
Adelie,Biscoe,34.5,18.1,187,2900,FEMALE
Adelie,Biscoe,41.4,18.6,191,3700,MALE
Adelie,Biscoe,39,17.5,186,3550,FEMALE
Chinstrap,Dream,50,19.5,196,3900,MALE
Chinstrap,Dream,51.3,19.2,193,3650,MALE
Chinstrap,Dream,45.4,18.7,188,3525,FEMALE
Chinstrap,Dream,52.7,19.8,197,3725,MALE
Chinstrap,Dream,45.2,17.8,198,3950,FEMALE
Chinstrap,Dream,46.1,18.2,178,3250,FEMALE
Chinstrap,Dream,51.3,18.2,197,3750,MALE
Chinstrap,Dream,46,18.9,195,4150,FEMALE
Chinstrap,Dream,51.3,19.9,198,3700,MALE
Chinstrap,Dream,46.6,17.8,193,3800,FEMALE
Chinstrap,Dream,51.7,20.3,194,3775,MALE
Chinstrap,Dream,47,17.3,185,3700,FEMALE
Chinstrap,Dream,52,18.1,201,4050,MALE
Chinstrap,Dream,45.9,17.1,190,3575,FEMALE
Chinstrap,Dream,50.5,19.6,201,4050,MALE
Chinstrap,Dream,50.3,20,197,3300,MALE
Chinstrap,Dream,58,17.8,181,3700,FEMALE
Chinstrap,Dream,46.4,18.6,190,3450,FEMALE
Chinstrap,Dream,49.2,18.2,195,4400,MALE
Chinstrap,Dream,42.4,17.3,181,3600,FEMALE
Chinstrap,Dream,48.5,17.5,191,3400,MALE
Chinstrap,Dream,43.2,16.6,187,2900,FEMALE
Chinstrap,Dream,50.6,19.4,193,3800,MALE
Chinstrap,Dream,46.7,17.9,195,3300,FEMALE
Chinstrap,Dream,52,19,197,4150,MALE
Chinstrap,Dream,50.5,18.4,200,3400,FEMALE
Chinstrap,Dream,49.5,19,200,3800,MALE
Chinstrap,Dream,46.4,17.8,191,3700,FEMALE
Chinstrap,Dream,52.8,20,205,4550,MALE
Chinstrap,Dream,40.9,16.6,187,3200,FEMALE
Chinstrap,Dream,54.2,20.8,201,4300,MALE
Chinstrap,Dream,42.5,16.7,187,3350,FEMALE
Chinstrap,Dream,51,18.8,203,4100,MALE
Chinstrap,Dream,49.7,18.6,195,3600,MALE
Chinstrap,Dream,47.5,16.8,199,3900,FEMALE
Chinstrap,Dream,47.6,18.3,195,3850,FEMALE
Chinstrap,Dream,52,20.7,210,4800,MALE
Chinstrap,Dream,46.9,16.6,192,2700,FEMALE
Chinstrap,Dream,53.5,19.9,205,4500,MALE
Chinstrap,Dream,49,19.5,210,3950,MALE
Chinstrap,Dream,46.2,17.5,187,3650,FEMALE
Chinstrap,Dream,50.9,19.1,196,3550,MALE
Chinstrap,Dream,45.5,17,196,3500,FEMALE
Chinstrap,Dream,50.9,17.9,196,3675,FEMALE
Chinstrap,Dream,50.8,18.5,201,4450,MALE
Chinstrap,Dream,50.1,17.9,190,3400,FEMALE
Chinstrap,Dream,49,19.6,212,4300,MALE
Chinstrap,Dream,51.5,18.7,187,3250,MALE
Chinstrap,Dream,49.8,17.3,198,3675,FEMALE
Chinstrap,Dream,48.1,16.4,199,3325,FEMALE
Gentoo,Biscoe,50,16.3,230,5700,MALE
Gentoo,Biscoe,48.7,14.1,210,4450,FEMALE
Gentoo,Biscoe,50,15.2,218,5700,MALE
Gentoo,Biscoe,47.6,14.5,215,5400,MALE
Gentoo,Biscoe,46.5,13.5,210,4550,FEMALE
Gentoo,Biscoe,45.4,14.6,211,4800,FEMALE
Gentoo,Biscoe,46.7,15.3,219,5200,MALE
Gentoo,Biscoe,43.3,13.4,209,4400,FEMALE
Gentoo,Biscoe,46.8,15.4,215,5150,MALE
Gentoo,Biscoe,40.9,13.7,214,4650,FEMALE
Gentoo,Biscoe,49,16.1,216,5550,MALE
Gentoo,Biscoe,45.5,13.7,214,4650,FEMALE
Gentoo,Biscoe,48.4,14.6,213,5850,MALE
Gentoo,Biscoe,45.8,14.6,210,4200,FEMALE
Gentoo,Biscoe,49.3,15.7,217,5850,MALE
Gentoo,Biscoe,42,13.5,210,4150,FEMALE
Gentoo,Biscoe,49.2,15.2,221,6300,MALE
Gentoo,Biscoe,46.2,14.5,209,4800,FEMALE
Gentoo,Biscoe,48.7,15.1,222,5350,MALE
Gentoo,Biscoe,50.2,14.3,218,5700,MALE
Gentoo,Biscoe,45.1,14.5,215,5000,FEMALE
Gentoo,Biscoe,46.5,14.5,213,4400,FEMALE
Gentoo,Biscoe,46.3,15.8,215,5050,MALE
Gentoo,Biscoe,42.9,13.1,215,5000,FEMALE
Gentoo,Biscoe,46.1,15.1,215,5100,MALE
Gentoo,Biscoe,47.8,15,215,5650,MALE
Gentoo,Biscoe,48.2,14.3,210,4600,FEMALE
Gentoo,Biscoe,50,15.3,220,5550,MALE
Gentoo,Biscoe,47.3,15.3,222,5250,MALE
Gentoo,Biscoe,42.8,14.2,209,4700,FEMALE
Gentoo,Biscoe,45.1,14.5,207,5050,FEMALE
Gentoo,Biscoe,59.6,17,230,6050,MALE
Gentoo,Biscoe,49.1,14.8,220,5150,FEMALE
Gentoo,Biscoe,48.4,16.3,220,5400,MALE
Gentoo,Biscoe,42.6,13.7,213,4950,FEMALE
Gentoo,Biscoe,44.4,17.3,219,5250,MALE
Gentoo,Biscoe,44,13.6,208,4350,FEMALE
Gentoo,Biscoe,48.7,15.7,208,5350,MALE
Gentoo,Biscoe,42.7,13.7,208,3950,FEMALE
Gentoo,Biscoe,49.6,16,225,5700,MALE
Gentoo,Biscoe,45.3,13.7,210,4300,FEMALE
Gentoo,Biscoe,49.6,15,216,4750,MALE
Gentoo,Biscoe,50.5,15.9,222,5550,MALE
Gentoo,Biscoe,43.6,13.9,217,4900,FEMALE
Gentoo,Biscoe,45.5,13.9,210,4200,FEMALE
Gentoo,Biscoe,50.5,15.9,225,5400,MALE
Gentoo,Biscoe,44.9,13.3,213,5100,FEMALE
Gentoo,Biscoe,45.2,15.8,215,5300,MALE
Gentoo,Biscoe,46.6,14.2,210,4850,FEMALE
Gentoo,Biscoe,48.5,14.1,220,5300,MALE
$\endgroup$
2
$\begingroup$

Have a look at the weights of your model at each step and the gradients that are being applied. In many cases the gradients are of order 10^-10 or smaller, meaning that the weights of the model basically do not change at all. The reason for this is that a neural network is sensitive to the scale of the data. It is therefore often good practice to scale your input variables, e.g. on a 0-1 scale. Simply dividing each column in the input by their max value using X_train /= X_train.max(axis=0) allows me to reach 90%+ training accuracy after 100 epochs (depending on the initialization of the weights). You can take the scaling even further by using something like MinMaxScaler or StandardScaler from scikit-learn.

$\endgroup$
1
  • $\begingroup$ Thanks for the suggestion. I used StandardScaler(). This improved my testing accuracy to 0.9565217391304348. However, my predictions were only right for 2 out of the 3 efforts. I scaled the training set, the test set, and even the single samples I was using to test the predictions. $\endgroup$
    – rsrobbins
    Jan 22 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.