1
$\begingroup$

I'm building PoS tagger for our language. I give tokens to the words and tags using Tokenizer(). Functions for word and tag are different.

# encode Y
tag_tokenizer = Tokenizer()
tag_tokenizer.fit_on_texts(tags)
Y_encoded = tag_tokenizer.texts_to_sequences(tags)

I have simple bidirectional LSTM to predict the algorithm.

bidirect_model = Sequential()
bidirect_model.add(Embedding(input_dim     = VOCABULARY_SIZE,
                             output_dim    = EMBEDDING_SIZE,
                             input_length  = MAX_SEQ_LENGTH,
                             trainable     = False
))
bidirect_model.add(Bidirectional(LSTM(64, return_sequences=True)))
bidirect_model.add(TimeDistributed(Dense(NUM_CLASSES, activation='softmax'))
bidirect_model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['acc'])
bidirect_training = bidirect_model.fit(Xtrain, ytrain, batch_size=128, epochs=10, validation_data=(Xvalidation, yvalidation))

loss, accuracy = bidirect_model.evaluate(Xtest, ytest, verbose = 1)
print("Loss: {0},\nAccuracy: {1}".format(loss, accuracy*100))

The result is pretty good, the accuracy is more than 94%. But when it comes to actually change the output from tokens to words it's a little struggle.

tag_tokenizer.sequences_to_texts(ytest.argmax(2))

When I try to change the ytest to text it works just fine but when I use the same method with prediction it doesn't convert it correctly.

ypred = bidirect_model.predict(Xtest)
tag_tokenizer.sequences_to_texts(ypred.argmax(2))

it returns the correct amount of the tags but it can return only one tag name for all of them. (For example noun for each tag, but there should be different PoS tags as well). I noticed that when I predict using LSTM the data actually changes to different format:

This is ypred:

array([[[9.96039510e-01, 3.83606239e-05, 3.69549962e-05, ...,
         5.24025600e-05, 9.90346816e-05, 4.23052406e-04],
        [9.97439384e-01, 3.10516843e-05, 2.83600530e-05, ...,
         3.12017110e-05, 5.57345738e-05, 2.51515419e-04],
        [9.98397648e-01, 2.60055913e-05, 2.21154296e-05, ...,
         1.76890717e-05, 2.89566851e-05, 1.39188676e-04],
        ...,
        [1.67124510e-01, 2.64671326e-01, 1.25262424e-01, ...,
         4.61455435e-03, 3.27696186e-03, 2.74900091e-03],
        [1.39361277e-01, 2.67008126e-01, 1.36608765e-01, ...,
         4.63735918e-03, 3.30406637e-03, 2.69036856e-03],

But actually my ytest is:

array([[[1., 0., 0., ..., 0., 0., 0.],
        [1., 0., 0., ..., 0., 0., 0.],
        [1., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 1., 0., ..., 0., 0., 0.],
        [0., 0., 1., ..., 0., 0., 0.],
        [0., 0., 1., ..., 0., 0., 0.]],

       [[1., 0., 0., ..., 0., 0., 0.],
        [1., 0., 0., ..., 0., 0., 0.],
        [1., 0., 0., ..., 0., 0., 0.],

I tried to round the numbers but that didn't work as well.

$\endgroup$
0
$\begingroup$

It appears that ypred is probability and ytest is one-hot encoded category. The most common way to convert probability to a single category prediction is softmax.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.