# Problems with Concatenating Embedded Categorical and Numerical variables for LSTM use

I am new to here and new to Deep Learning too, so apologies in advance for any ill formatted code or wordings. I have a data set where I track 4 variables with 2 categorical and 3 numerical fields, over 4 time steps.

I was able build a dataframe like this:

1. cat1 - SckitLearn Label Encoded variable names
2. cat2 - SckitLearn Label Encoded time steps
3. num1 : num3 - SckitLearn Normalized variables

I got it turn into a list of numpy arrays like this:

I want to embed and concatenate all these fields before feeding it into a LSTM in Keras, using this function:

def build_concat(df):
global inputs
inputs = []
global embeddings
embeddings = []
cat_cols = df.filter(like='cat')
num_cols = df.filter(like='num')
for cat_col in cat_cols:
cat_input = Input(shape=(4,4), name=cat_col)
unique_cat = cat_cols[cat_col].nunique()
embedding_size = min(np.ceil((unique_cat)/2), 20)
embedding_size = int(embedding_size)
cat_dim = unique_cat + 1
inputs.append(cat_input)
embeddings.append(Embedding(cat_dim, embedding_size, input_length = (4,4),
name=str(cat_col) + "_emb")(cat_input))
for num_col in num_cols:
num_input = Input(shape=(4,4), name=num_col)
inputs.append(num_input)
embeddings.append(num_input)


But I get this error :

ValueError: Shape must be rank 3 but is rank 2 for '{{node so2_model/concat/concat}} = ConcatV2[N=5, T=DT_FLOAT, Tidx=DT_INT32](so2_model/cat1_emb/embedding_lookup/Identity_1, so2_model/cat2_emb/embedding_lookup/Identity_1, IteratorGetNext:2, IteratorGetNext:3, IteratorGetNext:4, so2_model/concat/concat/axis)' with input shapes: [2,4,2], [2,4,2], [2,4], [2,4], [2,4], [].

This is my embedding list :

[<KerasTensor: shape=(None, 4, 4, 2) dtype=float32 (created by layer 'cat1_emb')>, <KerasTensor: shape=(None, 4, 4, 2) dtype=float32 (created by layer 'cat2_emb')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num1')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num2')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num3')>]

This is my input list, before embedding :

[<KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'cat1')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'cat2')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num1')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num2')>, <KerasTensor: shape=(None, 4, 4) dtype=float32 (created by layer 'num3')>]

You should reshape your numerical layers so that they have a shape (None,4,4,1). To concatenate, you need the all but one axis to be equal.

If you use:

tf.keras.layers.Reshape((4,4,1))(num_input)


this should work.

• Thanks! :) However I found that I have a different issue now. I made a smaller data set to experiment on Embeddings. I only took 1 cat variable with 4 levels and 1 numerical variable. I embedded the cat layer with cat dimension = 5 (I have 4 levels) and embedding size = 2. I found that the embedded output = (None,4,1,2) and the numerical variable output = (None,4,1). I am trying to concatenate these two, but I don't know how to. I need to input all these into an LSTM, so I need to preserve the input format in 3D.
– LHS
Apr 6 '21 at 20:13
• add an extra 1: python tf.keras.layers.Reshape((4,4,1,1))(num_input)  Apr 7 '21 at 8:07