I'm performing an aspect-based sentiment on consumer complaints. I'm tokenizing at the sentence level.
tidy_complaints <- tidy_complaints %>% unnest_tokens( output = sentence, input = consumer_complaint_narrative, token = 'sentences' ) %>% mutate(sentence_id = 1:n())
However, some of the the complaints contains words that are joined together such as "trailcertified" or "creditorcompany".
Is there any need to break this words up into meaningful words? if yes, how do I accomplish this and still keep the sentence intact?