I am having several features as text and I'm using them once for a classification problem and then later for a regression problem.

The textual data features themselves aren't categorical. Because as far I understand even though we generally classify features as either numerical or categorical but the text features do not have a limited set of values.

So I have two questions mainly:

  1. There are several NaN values, so should I really replace them with the most frequent value or replace by an empty string? The most frequent value occurs some 200 times out of a total no of values of 20k in each of the three features.

  2. I initially used TfidfTransformer and TfidfVectoriser to convert each feature to vector. However, each vector became very big in size. In figures of 19k - 30k. Each vector of different sizes. Then later I used HashVectoriser and converted each to a vector size of 1000 vectors. However, my classification problem still had a very low accuracy. So how do I identify what is the ideal size of vectors? Or should i try doing something else like combining the three features of each row and then vectorise them?

I'm kind of new to this, so thank you!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.