I have a dataset (~52k rows) with a column containing just pure sentences (upper and lowercase, with punctuation and stop words) in each row. What can I do to represent this data in a meaningful way for my model. I tried using a CountVectorizer but it's giving me a dataframe with way too many columns and the model is having a hard time using it. What other options do I have?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ What do you want to do with the dataset? Depends on the model too. For example, Gensim's LDA pretty much takes data $\endgroup$– Dawny33Mar 23, 2018 at 12:53
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
That is the real-world! in text analysis you usually end up with a extremely high-dimensional representation. First of all, I always recommend TF-IDF over original BoW (CountVectrizer). To end up with a low-dimensional space, you may use this naive approach which is not written efficient in computational way but the approach gives you an idea where you need to go. It will be more clear when I add explanations to the notebook soon.
answered May 22, 2018 at 17:52
$\begingroup$
$\endgroup$
There's a myriad of approaches you can do here but each really depend on your end goal. You can use a BOW approach and create a large sparse matrix like you've seen before with CountVectorizer or TfidfVectorizer. You can computer similarity scores between sentences using cosine distances. You can score them using bm25+. You can show average idf score. You can show co-occurance of terms in sentence when compared to values in same target or different target or all targets. You can embed the words into a continious vector space using word2vec, fasttext etc.
In the end, it really comes down to what you're overall goal.
$\begingroup$
$\endgroup$
I tried using a CountVectorizer but it's giving me a dataframe with way too many columns and the model is having a hard time using it
What model did you use? There are models that are well suited for such data, and some even can handle sparse data natively - in scikit-learn you can use :
LASSO or other sparse linear models.
Naive Bayes - Multinomial or Bernoulli one, make sure to try both, sometimes Bernoulli (which is simple, it only uses indicator, and not counts) performs better
Other than that, Factorization Machines can be also used (FMs inventor, Steffan Rendle did even win a Kaggle competition which had really sparse data using them).
Also another approach would be to just extract keywords and run BoW on them - for example Gensim library has methods to do that.
For both of the above approaches the suggestion mentioned by Kasra Manshaei is still valid.
Yet another approach would be to use neural networks - especially Recurrent NNs, but their usage is not that straightforward, and they might need serious computational resources to be trained reasonably (though they can use pretrained word embeddings, which makes it easier).
BTW if you have problems with keeping all that data in memory, you might try to dump it to disk and then use a library that gets files as input - for example Vowpal Wabbit.
$\begingroup$
$\endgroup$
I recommend that you use word2vec to represent a word as a vector. After that you can feed your sequence of word vectors to LSTM to get vector representation of your sentence or paragraph.
