Problem Statement: I have problem making the Entity Embedding of Categorical Variable works for a simple dataset. I have followed the original github, or paper, or other blogposts[1,2,or this 3], or this Kaggle kernel; still not working.
Data Part: I am using the Ames Housing dataset as was hosted in Kaggle. I'm loading it in pandas dataframe as:
url = 'http://www.amstat.org/publications/jse/v19n3/decock/AmesHousing.xls' # Load the file into a Pandas DataFrame data_df = pd.read_excel(url)
For simplicity, out of 81 independent features, I am ONLY taking the Neighborhood, which is categorical, and the Gr Liv Area, which is numerical. And SalePrice, which is our target. I also split the data into train, and test and normalize the numerical variables.
features = ['Neighborhood','Gr Liv Area'] target = ['SalePrice'] data_df=data_df[features + target] X_train, y_train = data_df.iloc[:2000][features], data_df.iloc[:2000][target] X_test = data_df.iloc[2000:][features] X_train['Gr Liv Area']=StandardScaler().fit_transform(X_train['Gr Liv Area'].reshape(-1, 1)) y_train=StandardScaler().fit_transform(y_train)
Embedding Neural Net: Here is the block of code where I am building the Entity Embedding Neural Net including both the categorical and numerical variables. In Entity Embedding, there is a particular hyperparamter that defines the embedding size (as we have in NLP). Here I am using of the above-mentioned blogpost strategy to choose that.
input_models= output_embeddings= numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64'] for categorical_var in X_train.select_dtypes(include=['object']): #Name of the categorical variable that will be used in the Keras Embedding layer cat_emb_name= categorical_var.replace(" ", "")+'_Embedding' # Define the embedding_size no_of_unique_cat = X_train[categorical_var].nunique() embedding_size = int(min(np.ceil((no_of_unique_cat)/2), 50 )) vocab = no_of_unique_cat+1 #One Embedding Layer for each categorical variable input_model = Input(shape=(1,)) output_model = Embedding(vocab, embedding_size, name=cat_emb_name)(input_model) output_model = Reshape(target_shape=(embedding_size,))(output_model) #Appending all the categorical inputs input_models.append(input_model) #Appending all the embeddings output_embeddings.append(output_model) #Other non-categorical data columns (numerical). #I define single another network for the other columns and add them to our models list. input_numeric = Input(shape=(len(X_train.select_dtypes(include=numerics).columns.tolist()),)) embedding_numeric = Dense(64)(input_numeric) input_models.append(input_numeric) output_embeddings.append(embedding_numeric) #At the end we concatenate altogther and add other Dense layers output = Concatenate()(output_embeddings) output = Dense(500, kernel_initializer="uniform")(output) output = Activation('relu')(output) output = Dense(256, kernel_initializer="uniform")(output) output = Activation('relu')(output) output = Dense(1, activation='sigmoid')(output) model = Model(inputs=input_models, outputs=output) model.compile(loss='mean_squared_error', optimizer='Adam',metrics=['mse','mape'])
At the end, the model looks like this:
This look OK to me, unless I'm missing sth. Anyway, when I'm training the model like below:
history = model.fit(X_train,y_train , epochs = 200 , batch_size = 16, verbose= 2)
I get a rather usual keras error:
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([['NAmes', 0.31507361227175135], ['NAmes', -1.2242024755540366], ['NAmes', -0.3472201781480285], ...,
Then I looked more carefully at the original github or that Kaggle kernel, I noticed one has to convert the data to list format to match the network structure (still I am not sure I fully understand WHY!, see the preproc function there). Anyway, I convert my data to the list format like:
X_train_list =  for i,column in enumerate(X_train.columns.tolist()): X_train_list.append(X_train.values[..., [i]])
Now when trying to train once again this time using the list format of the data i.e. X_train_list:
history = model.fit(X_train_list,y_train , epochs = 200 , batch_size = 16, verbose= 2)
This time it starts with the first Epoch, then immediately stops with the following error:
ValueError: could not convert string to float: 'Mitchel'
It is rather obvious that it complains about one of the categories of the only Neighborhood variable that I have not encoded! Sure I have not, I thought that was the whole purpose of the Entity Embedding that the networks initiates a random embedding weights and learn the best embedding of that categorical variable during optimization of the target. Super confused!! Any help is much appreciated.