0
$\begingroup$

The length of human_vocab is 18377. The length of input X is 1000

I'm trying to run to_categorical

np.array(list(map(lambda x: to_categorical(x, num_classes=len(human_vocab)), X)))

Is this the same if i apply:

onehot_encoder = OneHotEncoder(sparse=False)
onehot_encoder.fit_transform(X)

the output of onehot_encode is (1000, 9739) shouldn't it be (1000,18377) ?

human_vocab contains the corpus data that X doesnt contain I'm trying to find the replacement of to_categorical due to memory issues to create hot vector

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

To answer your question:

the output of onehot_encode is (1000, 9739) shouldn't it be (1000,18377) ?

The OneHotEncoder instance will create a dimension per unique word seen in the training sample. Here you are only showing it 9739 different words at training so it does not need more dimensions to perform one hot encoding.

One way to have it accommodate the entire vocabulary is

onehot_encoder = OneHotEncoder(sparse=False, categories=human_vocab)
onehot_encoder.fit_transform(X)
Improve this answer
$\endgroup$
2
$\begingroup$

Yes, OneHotEncoder and keras.utils.to_categorical are one and the same thing where one being a class and the other being a function.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.