Text mining of big data using R Server

Question

I have a big table (150 Million rows and ~ 70 columns). In three of the columns in the table I have text input (3-20 words/column), which I need to use for a classification algorithm.

For smaller datasets, I have used the tm R package and created a DocumentTermMatrix, where I used the frequency of word (or word parts) as predictors in a SVM & Decision Forest algorithm.

However, now my dataset is much bigger, which is why I converted it to a xdf file and used RevoScaleR packages for merging, joining etc. so far.

However, I am unsure how to do text mining with very large datasets (e.g., create a document term matrix which I can use for classification). I did not find any pre-build functions. It may be possible to create a document term matrix in a chunk of data, and then somehow sum the frequencies. I am not sure if that is the best way, could you help me with that?

Thank you, Lisa

Answer 1

Thank you for your reply. By accident I found out that MicrosoftML (https://msdn.microsoft.com/en-us/microsoft-r/microsoftml-introduction) offers just what I need - by using the featurizeText functions offered within Microsoft R client and R server. Maybe it helps others - I found their example and could it translate to my data (featurizeText() help under Microsoft ML).

trainReviews <- data.frame(review = c( 
        "This is great",
        "I hate it",
        "Love it",
        "Do not like it",
        "Really like it",
        "I hate it",
        "I like it a lot",
        "I kind of hate it",
        "I do like it",
        "I really hate it",
        "It is very good",
        "I hate it a bunch",
        "I love it a bunch",
        "I hate it",
        "I like it very much",
        "I hate it very much.",
        "I really do love it",
        "I really do hate it",
        "Love it!",
        "Hate it!",
        "I love it",
        "I hate it",
        "I love it",
        "I hate it",
        "I love it"),
     like = c(TRUE, FALSE, TRUE, FALSE, TRUE,
        FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE,
        FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, 
        FALSE, TRUE, FALSE, TRUE), stringsAsFactors = FALSE
    )

    testReviews <- data.frame(review = c(
        "This is great",
        "I hate it",
        "Love it",
        "Really like it",
        "I hate it",
        "I like it a lot",
        "I love it",
        "I do like it",
        "I really hate it",
        "I love it"), stringsAsFactors = FALSE)


outModel <- rxLogisticRegression(like ~ reviewTran, data = trainReviews,
    mlTransforms = list(featurizeText(vars = c(reviewTran = "review"),
    stopwordsRemover = stopwordsDefault(), keepPunctuations = FALSE)))
# 'hate' and 'love' have non-zero weights summary(outModel)

Answer 2

Do you mean 150 million rows and 70 columns?

Yeah, it is easy to run out of memory creating the DTMs. This is a similar question to here:

Text Mining on Large Dataset

Like they talk about, I would create in splits (batches) and save the chunks out to a file or a DB. When it is time to do classification, see how it works if you pull 10,000 randomly sampled rows.