I have a big table (150 Million rows and ~ 70 columns). In three of the columns in the table I have text input (3-20 words/column), which I need to use for a classification algorithm.
For smaller datasets, I have used the tm R package and created a DocumentTermMatrix, where I used the frequency of word (or word parts) as predictors in a SVM & Decision Forest algorithm.
However, now my dataset is much bigger, which is why I converted it to a xdf file and used RevoScaleR packages for merging, joining etc. so far.
However, I am unsure how to do text mining with very large datasets (e.g., create a document term matrix which I can use for classification). I did not find any pre-build functions. It may be possible to create a document term matrix in a chunk of data, and then somehow sum the frequencies. I am not sure if that is the best way, could you help me with that?
Thank you, Lisa