1
$\begingroup$

I have created a simple Keras deep learning model in python. Total no of variables in training are 195 while in unseen test data are 181.All input fields are categorical(converted by one hot encoding). Since unseen test data has some different categories thats why after one hot encoding fields are not matching with train.

So during predict step on unseen test data, model gives the following error. Is there any way out?

ValueError: Error when checking input: expected dense_30_input to have shape (195,)
but got array with shape (181,)
$\endgroup$
1
$\begingroup$

As others before me pointed out you should have exactly the same variables in your test data as in your training data.

In case of one-hot encoding if you have unseen categories in your test data your model doesn't know how to handle them it was not trained on those variables. In that case during data preparation you shall create all the variables that you had during training with the value of 0 and you don't create new variable for the unseen category.

I think your confusion and the differing number of variables come from the function that you use to do the one-hot encoding for you. Probably you run them on the two datasets separately and it will only create the variables that it founds in the specific datasets. You can overcome on it by using label encoder or onehotencoder transformer from scikit-learn that will save inside its obeject the original state and in every transformation it will recreate exactly the same structure.

UPDATE to use sklearn onehotencoder:

from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(handle_unknown='ignore')
encoder.fit(train_categorical_data)
encoded_train=encoder.transform(train_categorical_data)
encoded_test=encoder.transform(test_categorical_data)

You can save the encoder to use it later. See more about it in the official documentation.

$\endgroup$
  • $\begingroup$ ..Thanks for reply..Can you please share any example how we can overcome on this..how to implement using label encoder or onehotencoder transformer from scikit-learn? $\endgroup$ – Sarvendra Singh Jun 26 at 9:54
0
$\begingroup$

The model will only be able to predict when you have all the variables as training data. That's how model learnt and the weights of NNs got updated. You cannot predict with different categories when you didn't use them while training.

$\endgroup$
  • $\begingroup$ Thanks Sir for reply..but one question in real data any new column or any category in categorical field may come(for example in payment data any merchant can use new terminal no as new category) so Should our model not handle it on real-time? It can ignore new category and predict based on old categories on trained...I used H2o and h2o handles everything like any new field,new category and predict function never gives error ..it gives some probability....So any way we can put some checks to ignore new categories/new field before predicting ...this question puzzling me from long. $\endgroup$ – Sarvendra Singh Jun 21 at 11:42
0
$\begingroup$

As pointed out by others, test data should have the same variables as in your training data for the ML to work.

Here are my thoughts, if your training and test data categorical classes are not matching.

  1. can use sklearn onehotencoding and specify to ignore any unknown classes in test data.

    from numpy import array from sklearn.preprocessing import OneHotEncoder data = ['cold', 'cold', 'warm', 'cold', 'hot', 'hot', 'warm'] values = array(data) print(values) onehot_encoder = OneHotEncoder(sparse=False, handle_unknown='ignore') data2 = ['cold', 'warm', 'hot', 'colder'] values = values.reshape(-1, 1) train_encoded = onehot_encoder.fit_transform(values) values2 = array(data2).reshape(-1, 1) print(train_encoded) print(data2) print(onehot_encoder.transform(values2)) data3 = ['cold'] print(data3) values3 = array(data3).reshape(-1, 1) print(onehot_encoder.transform(values3))

output

`['cold' 'cold' 'warm' 'cold' 'hot' 'hot' 'warm']
[[1. 0. 0.]
 [1. 0. 0.]
 [0. 0. 1.]
 [1. 0. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]
['cold', 'warm', 'hot', 'colder']
[[1. 0. 0.]
 [0. 0. 1.]
 [0. 1. 0.]
 [0. 0. 0.]]
['cold']
[[1. 0. 0.]]`

2. or combine training and test data and perform one hot encoding to get all the class labels.

$\endgroup$
0
$\begingroup$

Use any publicly available data set which is good and clean and test your model output. You will get to know how the model predicts, if the model predicts well then you will get to know if there is an issue in your data or issue with your model.

$\endgroup$
  • $\begingroup$ issue is not with model..question is can we introduce new fields/new categories in unseen test data while predicting and still model gives prediction. $\endgroup$ – Sarvendra Singh Jun 21 at 11:45
  • $\begingroup$ lets consider we are preparing for an exam only for X subject. In examination hall we got the question paper for Y subject. Hopefully, won't able to answer and ideally get failed. In the same way model does. $\endgroup$ – Muralidhar A Jun 22 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.