0
$\begingroup$

I'm performing an aspect-based sentiment on consumer complaints. I'm tokenizing at the sentence level.

tidy_complaints <- tidy_complaints %>%
  unnest_tokens(
    output = sentence,
    input = consumer_complaint_narrative,
    token = 'sentences'
  ) %>%
  mutate(sentence_id = 1:n())

However, some of the the complaints contains words that are joined together such as "trailcertified" or "creditorcompany".

Is there any need to break this words up into meaningful words? if yes, how do I accomplish this and still keep the sentence intact?

$\endgroup$
1
1
$\begingroup$

The compound split is not trivial, but there are solutions that kind of work (as you can read on the link @Aditya shared).

Another way of dealing with the matter would be to tokenize the strings into character n-grams.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.