0
$\begingroup$

I am kind of new to neural networks, so I was doing some practice with iris data set. But the results are weird. Because I am running the network with the same data (not randomly chosen) and huge number of epochs (500 times), but results are sometimes 0,25 sometimes 1,0 or 0,95. The only thing that random is biases and weights. I think training the network with 500 epochs should remove this randomness at the end. But it seems like it doesn't or there is some another problem that I am not noticed yet. If you help me about this I will be thankful.

(PS: If you give me an advice about making the question better I will edit it, I am new at forum)

import tensorflow as tf
import pandas as pd
import numpy as np
from tqdm import tqdm

def to_one_hot(label_data):
    one_hot = []
    for item in label_data:
        temp = []
        for i in range(3):
            if(i == item):
                temp.append(1)
            else:
                temp.append(0)
        one_hot.append(temp)
    return one_hot

def read_data(n):
    df_data = pd.read_csv('iris.csv')
    x = []
    y = []
    species_dict = {'Iris-setosa' : 0,
            'Iris-versicolor' : 1,
            'Iris-virginica' : 2}
    for i in range(len(df_data['Id'])):
        temp_item = []
        temp_item.append(df_data['SepalLengthCm'][i])
        temp_item.append(df_data['SepalWidthCm'][i])
        temp_item.append(df_data['PetalLengthCm'][i])
        temp_item.append(df_data['PetalWidthCm'][i])
        x.append(temp_item)
        y.append(species_dict[df_data['Species'][i]])
    y = to_one_hot(y)
    x_train = []
    y_train = []
    x_test = []
    y_test = []
    for i in range(50):
        if (i < n):
            for t in range(3):
                x_train.append(x[i+50*t])
                y_train.append(y[i+50*t])
        else:
            for t in range(3):
                x_test.append(x[i+50*t])
                y_test.append(y[i+50*t])
    return x_train,x_test,y_train,y_test

x_train,x_test,y_train,y_test = read_data(40)

n_layers = 30
n_nodes = 7
n_classes = 3

x = tf.placeholder('float', [None,4])
y = tf.placeholder('float')

def neural_network_model(data):
    layers = []
    layers.append({'weights':tf.Variable(tf.random_normal([4,n_nodes])),
        'biases':tf.Variable(tf.random_normal([n_nodes]))})
    for i in range(n_layers-1):
        layers.append({'weights':tf.Variable(tf.random_normal([n_nodes,n_nodes])),
            'biases':tf.Variable(tf.random_normal([n_nodes]))})

    output_layer = {'weights':tf.Variable(tf.random_normal([n_nodes,n_classes])),
            'biases':tf.Variable(tf.random_normal([n_classes]))}

    for i in range(n_layers):
        if (i == 0):
            nl = tf.add(tf.matmul(data,layers[i]['weights']),layers[i]['biases'])
            nl = tf.nn.relu(nl)
        else:
            nl = tf.add(tf.matmul(nl,layers[i]['weights']),layers[i]['biases'])
            nl = tf.nn.relu(nl)

    output = tf.add(tf.matmul(nl,output_layer['weights']),output_layer['biases'])

    return output

def train_neural_network(x):
    prediction = neural_network_model(x)
    cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(logits = prediction, labels = y) )
    optimizer = tf.train.AdamOptimizer().minimize(cost)

    hm_epochs = 500

    with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())
        for epoch in tqdm(range(hm_epochs)):
            _ ,c = sess.run([optimizer,cost], feed_dict = {x:x_train, y:y_train})
        correct = tf.equal(tf.argmax(prediction,1),tf.argmax(y,1))
        accuracy = tf.reduce_mean(tf.cast(correct, 'float'))
        print('Accuracy : ',accuracy.eval({x:x_test,y:y_test}))

train_neural_network(x)
$\endgroup$
  • 1
    $\begingroup$ Try setting up some seeds first? $\endgroup$ – Aditya Aug 2 at 4:16
0
$\begingroup$

Neural network algorithms are stochastic.

This means they make use of randomness, such as initializing to random weights, and in turn the same network trained on the same data can produce different results.The random initialization allows the network to learn a good approximation for the function being learned. The most common form of randomness used in neural networks is the random initialization of the network weights. Although randomness can be used in other areas, here is just a short list:

  • Randomness in Initialization, such as weights.
  • Randomness in Regularization, such as dropout
  • Randomness in Layers, such as word embedding.
  • Randomness in Optimization, such as stochastic optimization.

The Solutions.
The are two main solutions.

Solution #1: Repeat Your Experiment

The traditional and practical way to address this problem is to run your network many times (30+) and use statistics to summarize the performance of your model, and compare your model to other models.

I strongly recommend this approach, but it is not always possible due to the very long training times of some models.

Solution #2: Seed the Random Number Generator

Alternately, another solution is to use a fixed seed for the random number generator.

from numpy.random import seed
seed(1)
| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your help $\endgroup$ – Baran SAHİN Aug 8 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.