Vectorizing text data for ML models

Question

Here is the sample data I have:

Tag 1(Val: X), Tag 2(Val: Y), Tag 3(Val: Z), Label (Val: P)

Tag 1(Val: A), Tag 2(Val: B), Tag 3(Val: C), Label (Val: Q)

Tag 1(Val: D), Tag 2(Val: E), Tag 3(Val: F), Label (Val: R)

Tag 1(Val: G), Tag 2(Val: H), Tag 3(Val: I), Label (Val: S)

All the values are strings and I need to encode them into vectors for training ML models using this data. How do I make sure that the strings are always converted to the "same" vectors everytime? I notice that when I try a test input with the same value as the training data, it gets vectorized into a different integer. What is the standard procedure for preserving the mapping between String <--> Hashed Integer representation so that i get the same hash everytime?

Answer 1

Regardless of the "hashing" algorithm that is used in your code, the same strings should be always mapped to unique values unless they only look the same but not really the same.

Please look for common errors such as capital vs non-capital letters, type of whitespaces (tab, blank, cr, ln), number of whitespaces, 1 vs l, 0 vs O, etc.

If this does not solve the issue, please provide some real examples (real tags with the different integer hashing) and a little bit more information about the code/algorithm that you used in your project.

Answer 2

Have you looked into a tokenizer? By your question, I can't tell if either (1) you don't know about tokenizing or (2) you are looking for an alternative to tokenizing.

I will assume the former and suggest that you read about tokenizing and go down that road. Keras provides a very simple approach to tokenizing and sequences that you can implement quickly and should get what you're looking for.