Keras model giving error when fields of unseen test data and train data are not same

Question

I have created a simple Keras deep learning model in python. Total no of variables in training are 195 while in unseen test data are 181.All input fields are categorical(converted by one hot encoding). Since unseen test data has some different categories thats why after one hot encoding fields are not matching with train.

So during predict step on unseen test data, model gives the following error. Is there any way out?

ValueError: Error when checking input: expected dense_30_input to have shape (195,)
but got array with shape (181,)

Answer 1

As others before me pointed out you should have exactly the same variables in your test data as in your training data.

In case of one-hot encoding if you have unseen categories in your test data your model doesn't know how to handle them it was not trained on those variables. In that case during data preparation you shall create all the variables that you had during training with the value of 0 and you don't create new variable for the unseen category.

I think your confusion and the differing number of variables come from the function that you use to do the one-hot encoding for you. Probably you run them on the two datasets separately and it will only create the variables that it founds in the specific datasets. You can overcome on it by using label encoder or onehotencoder transformer from scikit-learn that will save inside its obeject the original state and in every transformation it will recreate exactly the same structure.

UPDATE to use sklearn onehotencoder:

from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(handle_unknown='ignore')
encoder.fit(train_categorical_data)
encoded_train=encoder.transform(train_categorical_data)
encoded_test=encoder.transform(test_categorical_data)

You can save the encoder to use it later. See more about it in the official documentation.

Answer 2

The model will only be able to predict when you have all the variables as training data. That's how model learnt and the weights of NNs got updated. You cannot predict with different categories when you didn't use them while training.

Answer 3

As pointed out by others, test data should have the same variables as in your training data for the ML to work.

Here are my thoughts, if your training and test data categorical classes are not matching.

1. can use sklearn onehotencoding and specify to ignore any unknown classes in test data.

   from numpy import array from sklearn.preprocessing import OneHotEncoder data = ['cold', 'cold', 'warm', 'cold', 'hot', 'hot', 'warm'] values = array(data) print(values) onehot_encoder = OneHotEncoder(sparse=False, handle_unknown='ignore') data2 = ['cold', 'warm', 'hot', 'colder'] values = values.reshape(-1, 1) train_encoded = onehot_encoder.fit_transform(values) values2 = array(data2).reshape(-1, 1) print(train_encoded) print(data2) print(onehot_encoder.transform(values2)) data3 = ['cold'] print(data3) values3 = array(data3).reshape(-1, 1) print(onehot_encoder.transform(values3))

output

`['cold' 'cold' 'warm' 'cold' 'hot' 'hot' 'warm']
[[1. 0. 0.]
 [1. 0. 0.]
 [0. 0. 1.]
 [1. 0. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]
['cold', 'warm', 'hot', 'colder']
[[1. 0. 0.]
 [0. 0. 1.]
 [0. 1. 0.]
 [0. 0. 0.]]
['cold']
[[1. 0. 0.]]`

2. or combine training and test data and perform one hot encoding to get all the class labels.

Answer 4

Use any publicly available data set which is good and clean and test your model output. You will get to know how the model predicts, if the model predicts well then you will get to know if there is an issue in your data or issue with your model.