How to convert a text to lower case using tm package?

Question

I am using the below R code to convert text to lower case:

movie_Clean <- tm_map(movie_Clean, content_transformer(tolower))

However I end up getting the below error:

Error in FUN(content(x), ...) : invalid input 'I just wanna watch Jurassic World í ½í¸«' in 'utf8towcs'.

Please help how to overcome this error.

Answer 1

This seems like an encoding error. Try adding the line

Encoding(movie_Clean)  <- "UTF-8"

before you lowercase the data. Check out this answer for a little context: https://stackoverflow.com/a/28340080/4539807

Answer 2

There were many suggested solutions on stackoverflow that may have worked but not anymore in the newest versions of wordcloud and tm. The root cause is that the text data contain emoji characters which are encoded in surrogates that are messy. For more detailed explanation, please read this blog post that contains the solution.

The conversion that works in this case is

words <- iconv(words, "ASCII", "UTF-8", sub="byte")

Here's what failed:

I tried to convert the text BEFORE and AFTER creating the corpus with

words.corpus <- Corpus(VectorSource(words))

BEFORE:

Converting to UTF-8 on the text didn't work with:

words <- sapply(words, function(x) iconv(enc2utf8(x), sub = "byte"))

nor

for (i in 1:length(words))
{
    Encoding(words[[i]])="UTF-8"
}

AFTER:

Converting to UTF-8 on the corpus didn't work with:

words.corpus <- tm_map(words.corpus, removeWords, remove_words)

nor

words.corpus <- tm_map(words.corpus, content_transformer(stringi::stri_trans_tolower))

nor

words.corpus <- tm_map(words.corpus, function(x) iconv(x, to='UTF-8'))

nor

words.corpus <- tm_map(words.corpus, enc2utf8)

nor

words.corpus <- tm_map(words.corpus, tolower)

All these solutions may have worked at a certain point in time, so I don't want to discredit the authors. They may work some time in the future. But why they didn't work is almost impossible to say because there were good reasons why they were supposed to work. Anyway, just remember to convert the text before creating the corpus with:

words <- iconv(words, "ASCII", "UTF-8", sub="byte")