1
$\begingroup$

I'm working on my first NN following a tensorflow tut and trying to use my own data. After about 80 attempts of formatting my data and trying to load it into a dataset to train I'm throwing the towel.

Here is how my data currently looks

syslog_data = [
[302014,0,0,63878,30,3,1], [302014,0,0,3891,0,0,0], [302014,0,0,15928,0,0,2], [305013,5,0,123,99999,0,3],
[302014,0,0,5185,0,0,0], [305013,5,0,123,99999,0,3], [302014,0,0,56085,0,0,0], [110002,4,2,50074,99999,0,4],

In this the last item in each list is the label. If you can tell me if I need to reformat my data and how or just how to get it loaded into a dataset properly.

Thanks for any help or advice you can give

Here is the full code:

import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from . import syslog

print(tf.VERSION)
print(tf.keras.__version__)

model = tf.keras.Sequential()
# Adds a densely-connected layer with 64 units to the model:
model.add(layers.Dense(64, activation='relu'))
# Add another:
model.add(layers.Dense(64, activation='relu'))
# Add a softmax layer with 10 output units:
model.add(layers.Dense(10, activation='softmax'))

model.compile(optimizer=tf.train.AdamOptimizer(0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

dataset = tf.data.dataset.from_tensor_slices(syslog)

model.fit(dataset, epochs=10, steps_per_epoch=30)
$\endgroup$
  • $\begingroup$ WElcome to Data Science SE! Which tutorial did you follow? What error are you actually getting? Have you read the Keras documentation? Or the relevant Tensorflow docs for from_tensor_slices? $\endgroup$ – n1k31t4 Apr 25 at 19:15
  • $\begingroup$ Ive followed about 15 :/ but this one is the most relevant tensorflow.org/guide/keras I have received a number of errors from different attempts. the most recent is this - got shape [8972], but wanted [8972, 1]. from this code dataset = tf.data.Dataset.from_tensor_slices((data, labels)). Im pretty lost on what my training data should look like and how i should import it. $\endgroup$ – Alex F Apr 25 at 19:23
  • $\begingroup$ I can reformat as needed, I just dont know what to do $\endgroup$ – Alex F Apr 25 at 19:24
2
$\begingroup$

There are a couple of problems and things you might want to add to your existing script.

Below I separate your example data into two NumPy arrays:

  • input values x
  • labels y

It is also important to make sure they are of type float32, because Tensorflow will complain if you pass it integers (as they otherwise would be interpreted).

The following works for me, the model trains to completion:

import numpy as np
import tensorflow as tf
from tensorflow.keras import layers

syslog_data = [
    [302014, 0, 0, 63878, 30, 3, 1],
    [302014, 0, 0, 3891, 0, 0, 0],
    [302014, 0, 0, 15928, 0, 0, 2],
    [305013, 5, 0, 123, 99999, 0, 3],
    [302014, 0, 0, 5185, 0, 0, 0],
    [305013, 5, 0, 123, 99999, 0, 3],
    [302014, 0, 0, 56085, 0, 0, 0],
    [110002, 4, 2, 50074, 99999, 0, 4],
]

print(tf.VERSION)
print(tf.keras.__version__)

x = np.array([arr[:-1] for arr in syslog_data], dtype=np.float32)
y = np.array([arr[-1:] for arr in syslog_data], dtype=np.float32)

model = tf.keras.Sequential()
# Adds a densely-connected layer with 64 units to the model:
model.add(layers.Dense(64, activation="relu"))
# Add another:
model.add(layers.Dense(64, activation="relu"))
# Add a softmax layer with 10 output units:
model.add(layers.Dense(10, activation="softmax"))

model.compile(optimizer=tf.train.AdamOptimizer(0.001), loss="categorical_crossentropy", metrics=["accuracy"])

model.fit(x, y, epochs=10, steps_per_epoch=30)
$\endgroup$
  • 1
    $\begingroup$ I know we just met but I love you $\endgroup$ – Alex F Apr 25 at 19:37
1
$\begingroup$
import keras
import numpy as np
full_data = np.array(syslog_data)
X = full_data[:,:6]
Y = full_data[:,6]
# Convert labels to categorical one-hot encoding
one_hot_labels = keras.utils.to_categorical(Y, num_classes=10)

model.fit(X,Y, epochs=10, steps_per_epoch=30)

Does this work? I think I might be misunderstanding the problem.

$\endgroup$
  • $\begingroup$ I didnt under stand that it needed to be an array, thank you for replying $\endgroup$ – Alex F Apr 25 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.