breaking joined words into meaningful ones during text mining

Question

I'm performing an aspect-based sentiment on consumer complaints. I'm tokenizing at the sentence level.

tidy_complaints <- tidy_complaints %>%
  unnest_tokens(
    output = sentence,
    input = consumer_complaint_narrative,
    token = 'sentences'
  ) %>%
  mutate(sentence_id = 1:n())

However, some of the the complaints contains words that are joined together such as "trailcertified" or "creditorcompany".

Is there any need to break this words up into meaningful words? if yes, how do I accomplish this and still keep the sentence intact?

Answer 1

The compound split is not trivial, but there are solutions that kind of work (as you can read on the link @Aditya shared).

Another way of dealing with the matter would be to tokenize the strings into character n-grams.