I am having several features as text and I'm using them once for a classification problem and then later for a regression problem.
The textual data features themselves aren't categorical. Because as far I understand even though we generally classify features as either numerical or categorical but the text features do not have a limited set of values.
So I have two questions mainly:
There are several NaN values, so should I really replace them with the most frequent value or replace by an empty string? The most frequent value occurs some 200 times out of a total no of values of 20k in each of the three features.
I initially used TfidfTransformer and TfidfVectoriser to convert each feature to vector. However, each vector became very big in size. In figures of 19k - 30k. Each vector of different sizes. Then later I used HashVectoriser and converted each to a vector size of 1000 vectors. However, my classification problem still had a very low accuracy. So how do I identify what is the ideal size of vectors? Or should i try doing something else like combining the three features of each row and then vectorise them?
I'm kind of new to this, so thank you!