How to add more features in addition to a 100D word vector

Question

I am working on sentiment analysis using tweets text. I am able to build a word vector using Keras text_to_sequence() method with pretrained GloVe embeddings in Python.

The question is how to add extra features beside the created GloVe embeddings and shape it in right way to feed to the embedding layer and the algorithm in general? For example, assuming the embedding's length is 100 and the extra features are 3, will appending the 3 features to 100 vector resulting in 103 vector length is the right approach? If so how would you implement it.

Where:

corpusAsList is the corpus as text in a list.

maxlen is the maximum number of columns to pad the word sequence with, so that all vectors have the same length.

maxVocabulary is the maximum amount of words to be used in the sequence and ignore the rest of the words.

And tokenizer is used to represent a word as a digit in the vocabulary.

For instance, if we called texts_to_sequences() like this:

   # Create tokenized corpuss
    listTokenized = tokenizer.texts_to_sequences(corpusAsList)

    # Sequence dataset
    sequenceMatrixPrediction = sequence.pad_sequences(listTokenized, maxlen=maxlen)

And then trained a Bidirectional LSTM model like this:

def getLSTMModel(maxlen, maxVocabulary, embedSizeLSTM=100, weights=None):

    inp = Input(shape=(maxlen,))
    x = Embedding(maxVocabulary, embedSizeLSTM,
                 weights=[weights], trainable=False)(inp)
    x = Bidirectional(LSTM(80, return_sequences=True))(x)
    x = GlobalMaxPool1D()(x)
    x = Dropout(0.25)(x)
    x = Dense(70, activation="relu")(x)
    x = Dropout(0.25)(x)
    x = Dense(1, activation="sigmoid")(x)
    model = Model(inputs=inp, outputs=x)
    model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])


return model

How could we add three extra features to this model?

Answer 1

You can add a second Input layer to your architecture.

Look at this link Keras Guide Multiple Inputs for a more detailed explanation.

The output of the embedding layer can be combined with the secondary input and passed to the next layer.

auxiliary_input = Input(shape=(5,), name='aux_input')
x = keras.layers.concatenate([lstm_out, auxiliary_input])