1
$\begingroup$

I have finished my first NN (pretty exciting) and have started tweaking in hopes of improving results.

Epoch 1/10
30/30 [==============================] - 0s 9ms/step - loss: 69.2138 - acc: 
0.2937 - val_loss: 62.3838 - val_acc: 0.6250
Epoch 2/10
30/30 [==============================] - 0s 1ms/step - loss: 66.8716 - acc: 
0.3167 - val_loss: 64.4628 - val_acc: 0.1875
Epoch 3/10
30/30 [==============================] - 0s 1ms/step - loss: 65.6411 - acc: 
0.2750 - val_loss: 96.1742 - val_acc: 0.1250
Epoch 4/10
30/30 [==============================] - 0s 1ms/step - loss: 70.4931 - acc: 
0.2875 - val_loss: 91.4954 - val_acc: 0.2031
Epoch 5/10
30/30 [==============================] - 0s 1ms/step - loss: 68.8988 - acc: 
0.1813 - val_loss: 75.3798 - val_acc: 0.0000e+00
Epoch 6/10
30/30 [==============================] - 0s 1ms/step - loss: 69.1241 - acc: 
0.0000e+00 - val_loss: 77.7191 - val_acc: 0.0000e+00
Epoch 7/10
30/30 [==============================] - 0s 1ms/step - loss: 64.7226 - acc: 
0.3479 - val_loss: 74.3400 - val_acc: 0.5156
Epoch 8/10
30/30 [==============================] - 0s 1ms/step - loss: 68.5523 - acc: 
0.1719 - val_loss: 69.1414 - val_acc: 1.0000
Epoch 9/10
30/30 [==============================] - 0s 1ms/step - loss: 66.1609 - acc: 
0.7917 - val_loss: 70.9609 - val_acc: 0.8438
Epoch 10/10
30/30 [==============================] - 0s 1ms/step - loss: 68.6736 - acc: 
0.7552 - val_loss: 68.3616 - val_acc: 0.7344

Test-Accuracy: 0.4234375

What confuses me is how scattered and inconsistent my val_acc values are. Is this due to bad data or bad modeling?

Here is a snipit of my training data (It is syslog that i have parsed into key values)

syslog_data = [
[0.110002,0.4,0.2,0,0,0,5], [0.110002,0.4,0.2,0,0,0,5], [0.110002,0.4,0.1,0,0,0,5], [0.110002,0.4,0.2,0,0,0,5],
[0.302014,0,0,0,0.0,0,1], [0.302014,0,0,0,0.0,0,1], [0.302014,0,0,0,0.0,0,1], [0.302014,0,0,0,0.0,0,1],
[0.302014,0,0,0,0.0,0,1], [0.302014,0,0,0,0.0,0,1], [0.302014,0,0,0,0.0,0,1], [0.302014,0,0,0,0.0,0,1],
[0.419002,0.2,0.1,0,0,0,6], [0.419002,0.2,0.1,0,0,0,6], [0.419002,0.2,0.1,0,0,0,6], [0.419002,0.2,0.1,0,0,0,6],
[0.110002,0.4,0,0,0,0,5], [0.110002,0.4,0,0,0,0,5], [0.110002,0.4,0.1,0,0,0,5], [0.110002,0.4,0.2,0,0,0,5],
[0.305013,0.5,0,0,0,0,4], [0.305013,0.5,0,0,0,0,4], [0.305013,0.5,0,0,0,0,4], [0.305013,0.5,0,0,0,0,4],
[0.710003,0.1,0.1,0,0,0,8], [0.710003,0.1,0.1,0,0,0,8], [0.710003,0.1,0.1,0,0,0,8], [0.710003,0.1,0.1,0,0,0,8],
[0.302014,0,0,0,0.03,0.3,2], [0.302014,0,0,0,0.03,0.3,2], [0.302014,0,0,0,0.03,0.3,2], [0.302014,0,0,0,0.03,0.3,2],

And here is the code behind my NN

import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from syslog import syslog_data, syslog_eval, syslog_pred

print(tf.VERSION)
print(tf.keras.__version__)

x = np.array([arr[:-1] for arr in syslog_data], dtype=np.float32)
y = np.array([arr[-1:] for arr in syslog_data], dtype=np.float32)

x_ev = np.array([arr[:-1] for arr in syslog_eval], dtype=np.float32)
y_ev = np.array([arr[-1:] for arr in syslog_eval], dtype=np.float32)

model = tf.keras.Sequential()
# Adds a densely-connected layer with x units to the model:
model.add(layers.Dense(10, activation='relu', input_shape=(6,)))
# Add another:
model.add(layers.Dense(32, activation='relu'))
# Add another:
model.add(layers.Dense(32, activation='relu'))
# Add another:
model.add(layers.Dense(10, activation='relu'))
# Add a softmax layer with 8 output units:
model.add(layers.Dense(8, activation='softmax'))

dataset = tf.data.Dataset.from_tensor_slices((x, y))
dataset = dataset.batch(32).repeat()

val_dataset = tf.data.Dataset.from_tensor_slices((x_ev, y_ev))
val_dataset = val_dataset.batch(32).repeat()

model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

results = model.fit(dataset, epochs=10, steps_per_epoch=30,
          validation_data=val_dataset,
          validation_steps=2)

x = np.array([arr[:-1] for arr in syslog_pred], dtype=np.float32)
dataset = tf.data.Dataset.from_tensor_slices(x)

y = model.predict(x)
print(y)
print("Test-Accuracy:", np.mean(results.history["val_acc"]))

EDIT:

I have expanded my batch size to 2000 and get the following results:

Epoch 1/10
30/30 [==============================] - 1s 18ms/step - loss: 67.8644 - acc: 
0.1418 - val_loss: 62.3833 - val_acc: 0.0000e+00
Epoch 2/10
30/30 [==============================] - 0s 7ms/step - loss: 67.8003 - acc: 
0.0894 - val_loss: 64.4627 - val_acc: 0.0000e+00
Epoch 3/10
30/30 [==============================] - 0s 7ms/step - loss: 67.8639 - acc: 
0.2535 - val_loss: 96.1742 - val_acc: 1.0000
Epoch 4/10
30/30 [==============================] - 0s 7ms/step - loss: 67.8003 - acc: 
0.4979 - val_loss: 91.4954 - val_acc: 1.0000
Epoch 5/10
30/30 [==============================] - 0s 6ms/step - loss: 67.8639 - acc: 
0.1995 - val_loss: 75.3798 - val_acc: 0.0000e+00
Epoch 6/10
30/30 [==============================] - 0s 7ms/step - loss: 67.8003 - acc: 
0.4316 - val_loss: 77.7191 - val_acc: 0.0000e+00
Epoch 7/10
30/30 [==============================] - 0s 7ms/step - loss: 67.8639 - acc: 
0.4328 - val_loss: 74.3400 - val_acc: 0.0000e+00
Epoch 8/10
30/30 [==============================] - 0s 7ms/step - loss: 67.8002 - acc: 
0.7325 - val_loss: 69.1414 - val_acc: 0.0000e+00
Epoch 9/10
30/30 [==============================] - 0s 7ms/step - loss: 67.8639 - acc: 
0.8989 - val_loss: 70.9609 - val_acc: 1.0000
Epoch 10/10
30/30 [==============================] - 0s 7ms/step - loss: 67.8002 - acc: 
0.9663 - val_loss: 68.3616 - val_acc: 1.0000
[[0.12500003 0.12500003 0.12500001 0.12500003 0.125      0.12499999
  0.12499997 0.12499999]]
Test-Accuracy: 0.4
$\endgroup$
6
  • $\begingroup$ Do you only have 30 training examples? If so, you're trying to learn an extremely large number of parameters from a very small number of examples, and that may account for the erratic behavior of your network. $\endgroup$
    – Andy M
    Apr 26 '19 at 14:18
  • $\begingroup$ I have 8k, ill expand that and give the results $\endgroup$
    – Alex F
    Apr 26 '19 at 14:20
  • $\begingroup$ Added in edit with expanded batch $\endgroup$
    – Alex F
    Apr 26 '19 at 14:24
  • $\begingroup$ Are you setting steps_per_epoch for time concerns? Depending on the batch_size you choose you aren't seeing the whole dataset in each epoch. Can you try setting the batch_size to a moderate number (32 or something) and not specifying the steps_per_epoch during training? $\endgroup$
    – Andy M
    Apr 26 '19 at 14:29
  • $\begingroup$ ValueError: When using iterators as input to a model, you should specify the steps_per_epoch argument $\endgroup$
    – Alex F
    Apr 26 '19 at 14:33
0
$\begingroup$

Fixed it. It was a problem with my labels.

Epoch 1/10
100/100 [==============================] - 7s 66ms/step - loss: 2.0792 - acc: 0.2209 
- val_loss: 2.1252 - val_acc: 0.1250
Epoch 2/10
100/100 [==============================] - 6s 63ms/step - loss: 1.9951 - acc: 0.2380 
- val_loss: 1.7550 - val_acc: 0.3750
Epoch 3/10
100/100 [==============================] - 6s 64ms/step - loss: 1.1870 - acc: 0.6727 
- val_loss: 0.6140 - val_acc: 0.8750
Epoch 4/10
100/100 [==============================] - 6s 64ms/step - loss: 0.4538 - acc: 0.9728 
- val_loss: 0.3049 - val_acc: 1.0000
Epoch 5/10
100/100 [==============================] - 5s 55ms/step - loss: 0.2086 - acc: 1.0000 
- val_loss: 0.1202 - val_acc: 1.0000
Epoch 6/10
100/100 [==============================] - 6s 56ms/step - loss: 0.1060 - acc: 1.0000 
- val_loss: 0.0746 - val_acc: 1.0000
Epoch 7/10
100/100 [==============================] - 7s 69ms/step - loss: 0.0633 - acc: 1.0000 
- val_loss: 0.0493 - val_acc: 1.0000
Epoch 8/10
100/100 [==============================] - 7s 65ms/step - loss: 0.0423 - acc: 1.0000 
- val_loss: 0.0381 - val_acc: 1.0000
Epoch 9/10
100/100 [==============================] - 6s 59ms/step - loss: 0.0305 - acc: 1.0000 
- val_loss: 0.0218 - val_acc: 1.0000
Epoch 10/10
100/100 [==============================] - 6s 56ms/step - loss: 0.0232 - acc: 1.0000 
- val_loss: 0.0209 - val_acc: 1.0000
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.