I have a dataframe which has two columns of interest: A and B with string values. I am trying to build a prediction model which takes in a set of values in A as input and predicts the corresponding B values. I am trying to one-hot encode the string values before giving it to the neural network. This is what I have done:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import OneHotEncoder
from keras.models import Sequential
from keras.layers import Dense

def load_dataset(filename):
    # load the dataset as a pandas DataFrame
    data = pd.read_csv(filename, header=None)
    dataset = data.values
    X = dataset[:, :-1]
    y = dataset[:,-1]
    # reshape target to be a 2d array
    y = y.reshape((len(y), 1))
    return X, y

def prepare_inputs(X_train, X_test):
    ohe = OneHotEncoder()
    X_train_enc = ohe.transform(X_train)
    X_test_enc = ohe.transform(X_test)
    return X_train_enc, X_test_enc

def prepare_targets(y_train, y_test):
    ohe = OneHotEncoder()
    y_train_enc = ohe.transform(y_train)
    y_test_enc = ohe.transform(y_test)
    return y_train_enc, y_test_enc

# load the dataset
X, y = load_dataset('rdf.csv')

# split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1)

# prepare input data
X_train_enc, X_test_enc = prepare_inputs(X_train, X_test)

# prepare output data
y_train_enc, y_test_enc = prepare_targets(y_train, y_test)

# define the  model
model = Sequential()
model.add(Dense(32, input_dim=X_train_enc.shape[1], activation='relu', kernel_initializer='he_normal'))
model.add(Dense(32, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

# compile the keras model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# fit the keras model on the dataset
model.fit(X_train_enc, y_train_enc, epochs=100, batch_size=16, verbose=2)

# evaluate the keras model
_, accuracy = model.evaluate(X_test_enc, y_test_enc, verbose=0)

print('Accuracy: %.2f' % (accuracy*100))

But I am getting this error: ValueError: Found unknown categories [...]. Is there any efficient way to solve this usecase?


1 Answer 1


what I understand from your code is you are fitting a one-hot encoder on your training set, which may not include all words that appear in your test set. So when you get a new word in your evaluation method, your transformer cannot hash it, and hence throw an error.

the easiest way to solve this would be to use the unknown_error argument in one hot encoder

def prepare_targets(y_train, y_test):
  ohe = OneHotEncoder(handle_unknown='ignore')
  y_train_enc = ohe.transform(y_train)
  y_test_enc = ohe.transform(y_test)
  return y_train_enc, y_test_enc

now when you encounter a new word in the test set your encoder will output an array of all zeros, and if your try inverse transforms on all zeros, you will get none.

The downside is obvious here, your Neural Network does not what to do for zero vector as it has never seen such an example in training. So expect a weird response when a new word is evaluated.

The smart way of solving this problem would be to assume there are some unknown words in the training set by synthesizing few rare (unique) words as 'unknown' (replace words with the word 'unknown') and then do the one-hot encoding. while evaluating you will then first check if the word is not part of words set in training data, then you will replace that word with 'unknown'.

  • $\begingroup$ Thank you for getting back. I was able to solve that issue. But after that I am getting this error though: TypeError: Failed to convert object of type <class 'tensorflow.python.framework.sparse_tensor.SparseTensor'> to Tensor. Contents: SparseTensor(indices=Tensor("DeserializeSparse_1:0", shape=(None, 2), dtype=int64), values=Tensor("DeserializeSparse_1:1", shape=(None,), dtype=float32), dense_shape=Tensor("stack_1:0", shape=(2,), dtype=int64)). Consider casting elements to a supported type. $\endgroup$ Commented Apr 29, 2021 at 4:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.