I have a few word vectors-



Now i want to tokenized them and then make them into sequences to feed into the model-

For a standard word vector I would do something like this-

### Create sequence
vocabulary_size = 20000
tokenizer = Tokenizer(num_words= vocabulary_size)
sequences = tokenizer.texts_to_sequences(df['text'])
data = pad_sequences(sequences, maxlen=50)

But in my data I have non dictionary words and also I have some repeating words. How do I convert this data into sequences?

  • $\begingroup$ Can you share an e.g. of what you want to do? $\endgroup$
    – Aditya
    Commented Aug 2, 2020 at 4:16
  • $\begingroup$ Would something like sentencepiece(github.com/google/sentencepiece) work for you, which treats OOV? $\endgroup$
    – 20-roso
    Commented Aug 2, 2020 at 11:14

1 Answer 1


this can be done in multiple ways i am not one hundred percent understand the question because it is badly worded but there are 3 ways!

if you are doing this for a neural network you can use keras embedding layer and to create the sequence to feed to this embedding layer you can use one hot and padding from the preprocessing packages of course the sequence can be used regardless of if it is a nn the second thing you can do is use gensim to create word to vector the create a list of words or if you want the most control you can do something like

for i in input_texts:
    for char in i:
        if char not in input_characters:

for i in target_texts:
    for char in i:
        if char not in target_characters:
and then 

input_token_index = dict(
    [(char, i/num_encoder_tokens) for i, char in enumerate(input_characters)])
target_token_index = dict(
    [(char, i/num_decoder_tokens) for i, char in enumerate(target_characters)])

in your scenario you can just iterate through the sentence with a for loop and then create a dictionary after which you can feed it through a one hot and padding preprocessing from Keras.preprocessing and finally create an embedding


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.