# Confusion about Entity Embeddings of Categorical Variables - Working Example!

Problem Statement: I have problem making the Entity Embedding of Categorical Variable works for a simple dataset. I have followed the original github, or paper, or other blogposts[1,2,or this 3], or this Kaggle kernel; still not working.

Data Part: I am using the Ames Housing dataset as was hosted in Kaggle. I'm loading it in pandas dataframe as:

url = 'http://www.amstat.org/publications/jse/v19n3/decock/AmesHousing.xls'
# Load the file into a Pandas DataFrame


For simplicity, out of 81 independent features, I am ONLY taking the Neighborhood, which is categorical, and the Gr Liv Area, which is numerical. And SalePrice, which is our target. I also split the data into train, and test and normalize the numerical variables.

features = ['Neighborhood','Gr Liv Area']
target = ['SalePrice']
data_df=data_df[features + target]

X_train, y_train = data_df.iloc[:2000][features], data_df.iloc[:2000][target]
X_test = data_df.iloc[2000:][features]

X_train['Gr Liv Area']=StandardScaler().fit_transform(X_train['Gr Liv Area'].reshape(-1, 1))
y_train=StandardScaler().fit_transform(y_train)


Embedding Neural Net: Here is the block of code where I am building the Entity Embedding Neural Net including both the categorical and numerical variables. In Entity Embedding, there is a particular hyperparamter that defines the embedding size (as we have in NLP). Here I am using of the above-mentioned blogpost strategy to choose that.

input_models=[]
output_embeddings=[]
numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']

for categorical_var in X_train.select_dtypes(include=['object']):

#Name of the categorical variable that will be used in the Keras Embedding layer
cat_emb_name= categorical_var.replace(" ", "")+'_Embedding'

# Define the embedding_size
no_of_unique_cat  = X_train[categorical_var].nunique()
embedding_size = int(min(np.ceil((no_of_unique_cat)/2), 50 ))
vocab  = no_of_unique_cat+1

#One Embedding Layer for each categorical variable
input_model = Input(shape=(1,))
output_model = Embedding(vocab, embedding_size, name=cat_emb_name)(input_model)
output_model = Reshape(target_shape=(embedding_size,))(output_model)

#Appending all the categorical inputs
input_models.append(input_model)

#Appending all the embeddings
output_embeddings.append(output_model)

#Other non-categorical data columns (numerical).
#I define single another network for the other columns and add them to our models list.
input_numeric = Input(shape=(len(X_train.select_dtypes(include=numerics).columns.tolist()),))
embedding_numeric = Dense(64)(input_numeric)
input_models.append(input_numeric)
output_embeddings.append(embedding_numeric)

#At the end we concatenate altogther and add other Dense layers
output = Concatenate()(output_embeddings)
output = Dense(500, kernel_initializer="uniform")(output)
output = Activation('relu')(output)
output = Dense(256, kernel_initializer="uniform")(output)
output = Activation('relu')(output)
output = Dense(1, activation='sigmoid')(output)

model = Model(inputs=input_models, outputs=output)


At the end, the model looks like this:

This look OK to me, unless I'm missing sth. Anyway, when I'm training the model like below:

history  =  model.fit(X_train,y_train  , epochs =  200 , batch_size = 16, verbose= 2)


I get a rather usual keras error:

ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([['NAmes', 0.31507361227175135],
['NAmes', -1.2242024755540366],
['NAmes', -0.3472201781480285],
...,


Then I looked more carefully at the original github or that Kaggle kernel, I noticed one has to convert the data to list format to match the network structure (still I am not sure I fully understand WHY!, see the preproc function there). Anyway, I convert my data to the list format like:

X_train_list = []

for i,column in enumerate(X_train.columns.tolist()):
X_train_list.append(X_train.values[..., [i]])


Now when trying to train once again this time using the list format of the data i.e. X_train_list:

history  =  model.fit(X_train_list,y_train  , epochs =  200 , batch_size = 16, verbose= 2)


This time it starts with the first Epoch, then immediately stops with the following error:

ValueError: could not convert string to float: 'Mitchel'


It is rather obvious that it complains about one of the categories of the only Neighborhood variable that I have not encoded! Sure I have not, I thought that was the whole purpose of the Entity Embedding that the networks initiates a random embedding weights and learn the best embedding of that categorical variable during optimization of the target. Super confused!! Any help is much appreciated.

• very interesting topic! Have you been able to find a solution so far? – Seymour Jan 26 '19 at 23:08
• It is indeed interesting. Yes I have managed to make it work a few days after posting this questions here, but totally forgot to put it together and close the questions. I just cleaned up my notebook, and documented it and push everything in in a repository (see the link in the answer!). – TwinPenguins Jan 27 '19 at 11:51
• that is great news :) did Entity Embeddings improve your solution to the problem you were facing? are you satisfied? – Seymour Jan 27 '19 at 11:55
• Well, that requires more time to benchmark and compare. It may differ from one case to another. I guess it might be superior in datasets containing high-cardinal categorical features. What I like about this method, is that it is quite neat and simple, in contrast to the well-known one-hot encoding that has its complications. Let me know how it works for your case. – TwinPenguins Jan 27 '19 at 12:03
• I do not need to use it. I discovered its existence because a guy mentioned a medium article about this topic in the microsoft malware competition. I investigated a bit and it seemed a very cool tool. Hope one day it will come at hand :) – Seymour Jan 27 '19 at 12:06

• The fact is that you had to convert categories into a numerical features? So for example feature color = ["red", "blue", "green"] needs first to be converted into color_encoded = [0, 1, 2] ? – Seymour Jan 27 '19 at 12:03