0
$\begingroup$

I am having several features as text and I'm using them once for a classification problem and then later for a regression problem.

The textual data features themselves aren't categorical. Because as far I understand even though we generally classify features as either numerical or categorical but the text features do not have a limited set of values.

So I have two questions mainly:

  1. There are several NaN values, so should I really replace them with the most frequent value or replace by an empty string? The most frequent value occurs some 200 times out of a total no of values of 20k in each of the three features.

  2. I initially used TfidfTransformer and TfidfVectoriser to convert each feature to vector. However, each vector became very big in size. In figures of 19k - 30k. Each vector of different sizes. Then later I used HashVectoriser and converted each to a vector size of 1000 vectors. However, my classification problem still had a very low accuracy. So how do I identify what is the ideal size of vectors? Or should i try doing something else like combining the three features of each row and then vectorise them?

I'm kind of new to this, so thank you!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.