Text mining in R without RWeka

Question

I am following this tutorial to get a list of most recurrent n-word phrases from a dataframe that has approximately 2m records (text in multiple languages). Everything works fine till the last couple of steps

# tokenize into tri-grams
trigram.twitterTdm <- tm::TermDocumentMatrix(text.corpus["twitter"], control = list(tokenize = TrigramTokenizer))

where it ends up using TrigramTokenizer (from the RWeka package) to actually create the tokens. Issue is that RWeka has an rJava dependency which is not supported on our server.

Are there any other packages in R which I can use to get most recurrent 2/3/4 word phrases from a dataset? Any tutorials would be extremely useful (novice in this area)

Answer 1

Since you are using the tm library NLP should be installed as well and the code below should work. Adjust the tokenizer to your needs.

library(tm)
data("crude")
crude <- as.VCorpus(crude)
crude <- tm_map(crude, stripWhitespace)
crude <- tm_map(crude, removePunctuation)
crude <- tm_map(crude, content_transformer(tolower))
crude <- tm_map(crude, removeWords, stopwords("english"))
crude <- tm_map(crude, stemDocument)



# change 3 into whatever you need for a differenct n-gram
NLPtrigramTokenizer <- function(x) {
      unlist(lapply(ngrams(words(x), 3), paste, collapse = " "), use.names = FALSE)
}

tdm_NLP <- TermDocumentMatrix(crude, control=list(tokenize = NLPtrigramTokenizer))

inspect(tdm_NLP)
<<TermDocumentMatrix (terms: 2252, documents: 20)>>
Non-/sparse entries: 2485/42555
Sparsity           : 94%
Maximal term length: 32
Weighting          : term frequency (tf)
Sample             :
                       Docs
Terms                   144 236 237 242 246 248 273 489 502 704
  158 mln bpd             3   1   0   0   0   0   0   0   0   0
  ali alkhalifa alsabah   0   1   0   0   0   1   0   0   0   0
  arabian oil minist      0   0   0   0   0   1   1   0   0   0
  barrel per day          0   0   0   0   1   1   1   0   0   0
  decemb opec accord      0   0   0   0   0   1   1   0   0   0
  emerg opec meet         0   1   0   0   0   1   0   0   0   0
  hold futur posit        0   0   0   0   0   0   0   0   0   3
  industri sourc said     1   1   0   1   0   0   0   0   0   0
  sheikh abdulaziz said   0   0   0   0   4   0   0   0   0   0
  world oil price         0   1   1   0   1   1   0   0   0   0

Also you might be interested in reading Tidy text mining with R