0
$\begingroup$

I'm working on a project in which I should classify System calls sequences, my dataset is represented as sequences of integers (from 1 to 340). To do the classification I have inspired from Text classification projects. I'm trying to use on of them but I found a problem in my dataset shape, the code is:

df = pd.read_csv("data.txt") 
#df_test = pd.read_csv("validation.txt")

#split arrays into train and test data (cross validation)
train_text, test_text, train_y, test_y = train_test_split(df,df,test_size = 
0.2)

#train_text, train_y = (df,df)
#test_text, test_y = (df_test, df_test)
MAX_NB_WORDS = 5700

texts_train = train_text.astype(str)
texts_test = test_text.astype(str)


tokenizer = Tokenizer(nb_words=MAX_NB_WORDS, char_level=False)
tokenizer.fit_on_texts(texts_train)
sequences = tokenizer.texts_to_sequences(texts_train)
sequences_test = tokenizer.texts_to_sequences(texts_test)

word_index = tokenizer.word_index
#print('Found %s unique tokens.' % len(word_index))
type(tokenizer.word_index), len(tokenizer.word_index)
index_to_word = dict((i, w) for w, i in tokenizer.word_index.items()) 
" ".join([index_to_word[i] for i in sequences[0]])

seq_lens = [len(s) for s in sequences]
#print("average length: %0.1f" % np.mean(seq_lens))
#print("max length: %d" % max(seq_lens))

MAX_SEQUENCE_LENGTH = 100
# pad sequences with 0s
x_train = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)  # former des 
sequence de meme taille 150, en ajoutant des 0
x_test = pad_sequences(sequences_test, maxlen=MAX_SEQUENCE_LENGTH)
#print('Shape of data train:', x_train.shape)  #it gives (1,100)
#print('Shape of data test tensor:', x_test.shape)
y_train = train_y
y_test = test_y
#if np.any(y_train):
  #y_train = to_categorical(y_train)
print('Shape of label tensor:', y_train.shape)
EMBEDDING_DIM = 50
N_CLASSES = 2
# input: a sequence of MAX_SEQUENCE_LENGTH integers
sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='float32')

embedding_layer = Embedding(MAX_NB_WORDS, EMBEDDING_DIM,
                        input_length=MAX_SEQUENCE_LENGTH,
                        trainable=True)
embedded_sequences = embedding_layer(sequence_input)

average = GlobalAveragePooling1D()(embedded_sequences)
predictions = Dense(N_CLASSES, activation='softmax')(average)

   model = Model(sequence_input, predictions)
model.compile(loss='categorical_crossentropy',
          optimizer='adam', metrics=['acc'])
model.fit(x_train, y_train, validation_split=0.1,
      nb_epoch=10, batch_size=100)
output_test = model.predict(x_test)
print("test auc:", roc_auc_score(y_test,output_test[:,1]))

I got the error: ValueError Error when checking target expected dense_1 to have shape(2,), but got array with shape(1,)

Any suggestion, cause I don't know how to proceed . Thank you

$\endgroup$
0
$\begingroup$

The problem seems to arise in this line.

predictions = Dense(N_CLASSES, activation='softmax')(average) 

Here, N_CLASSES has the value of 2. Meaning you have 2 classes. But the y_train vector has shape ( 1, ).

  • You need to convert the y_train vector to a one hot vector. Here's any example.

If y_train = [ [ 0 ] , [ 1 ] ] then its one hot vector would be [ [ 1 , 0 ] , [ 0 , 1 ] ]. The one hot vector has shape ( 2 , ) which is needed in the above code snippet. So make this edit.

y_train = keras.utils.to_categorical( y_train , N_CLASSES )
$\endgroup$
  • $\begingroup$ Thanks for your answer. I tried this an I had this error: ValueError: Input arrays should have the same number of samples as target arrays. Found 1 input samples and 0 target samples $\endgroup$ – Kikio Feb 20 at 13:02
  • $\begingroup$ Make sure that you have the number of labels equal to the number of samples. $\endgroup$ – Shubham Panchal Feb 20 at 14:34
  • $\begingroup$ I have two classes which means there are two labels, or something else ? Sorry but I'm a beginner $\endgroup$ – Kikio Feb 20 at 18:00
  • $\begingroup$ No, you will count only the number of labels and not the number of one hot numbers $\endgroup$ – Shubham Panchal Feb 21 at 1:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.