Problem
I have data that includes multiple different text inputs as well as floats, categories, etc. Therefore I need to pass several different data types as features, including text which is an int array when tokenized.
Question
Say I tokenize the several text inputs; can I pass the tokenized text array as a feature alongside my floats and categories? If not, how is this done?
Background
When I've done NLP models, my code looks similar to this:
...
tokenizer = Tokenizer(num_words=max_features)
tokenizer.fit_on_texts(df['Stem'])
list_tokenized_train = tokenizer.texts_to_sequences(df['Stem'])
X_train = pad_sequences(list_tokenized_train, maxlen=10)
y_train = df['TotalPValue']
...
So, the text input becomes an array of int tokens padded with zeroes, e.g. [0 0 0 0 0 0 0 1 12 52].
This is not enough to solve my problem. I want to instead use multiple tokenized string inputs and floats as features. I want to first tokenize and pad each text input like above and put them in the same input array, like this: X_train = [[0 0 0 0 0 0 0 1 12 52], [0 0 0 0 0 0 0 42 12 23], 0.0425672].
I want to then start my model like this:
model = Sequential()
model.add(Embedding(max_features, embedding_vector_length, input_length=3))
Will it work if implemented like this?
My attempts
I searched for a while but couldn't find anyone else doing it like this. Surprising to me that I couldn't find anything since it seems like a basic problem.
Just wanted to know if I have the right idea, since - as a beginner - implementation will cost a lot of time if this isn't the right way of doing it. Thanks so much for the insight!
