2
$\begingroup$

I'm trying to build a regression model, in which one of the features contains text data. I was thinking in using scikit-learn's sklearn.feature_extraction.text.TfidfVectorizer. The issue however, is that the actual strings contain very few words. Exactly 1.8 in average. Here's a sample:

 print(df.keyword)

0 fre lifeproof
1 car stereo
2 analog clock
3 refrigerator

So my question is,

  • is the TfidfVectorizer also suited for such case? Or will such sparse resulting matrix not be benefiting the model?
  • Is there a more suited approach for such case?
$\endgroup$
0
$\begingroup$

Are the strings representing disjunct categories? If yes, you may want so use it as categorical feature for your model. If there are too much seperate categories you may want to group them into higher level categories such as refrigerator maps to kitchen products.

$\endgroup$
  • $\begingroup$ I can't really see how such mapping could be viable. Also there's another feature containing this higher level information. There are over 5000 distinct strings. There's also a higher level feature also representing the category of a given product. Such as furniture for sofa $\endgroup$ – yatu Sep 6 '19 at 10:44
0
$\begingroup$

It depends what it's going to be used for, but in general it can make sense to use TF-IDF with short sentence. The main difference with the more standard case of long sentences is that TF (Term Frequency) won't play any role since the frequency will almost always be 1. IDF can still be useful though, assuming it's relevant to assign more weight to rare words than to frequent words.

However the problem is that comparisons such as cosine are often going to be zero, since there will be little chance of words in common between two sentences.

$\endgroup$
0
$\begingroup$

As I see TF-IDF is very poor (more or less like a category feature) and does not take the words precedence (approximately 80% of cases have 2 or more words).

One possible approach is to do embedding vord2vect => Doc2Vect of every sentence. One good implementation is the gensim library: https://radimrehurek.com/gensim/models/doc2vec.html

doc2vect will deliver you a n-feature vector (the dimension of embedding doc2vect result) that can be used in your regression model.

Another approach more bizarre is to see what happens if you enter every sentence as a string of characters and enter this sequence into an LSTM layer that will deliver you also a n-feature result (the dimension of LSTM layer)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.