1
$\begingroup$

I am trying to analyze auto claims narrative documents which contain description about the accident usually free text written by claims executives. Is there a nlp technique I could use to identify cause of loss like : drunk and driving , negligence , bad weather , etc ?

I have used TF-IDF technique to rank order words per claim but it does really help to identify the exact cause of loss easily even if I concentrate on top 20% words. I am aware of word embedding but really not sure if they could help .

Also tried LDA topic modeling in the past but that gives topics for entire corpus rather than claim level.

Below is the code I used to calculate tf-idf scores for the unigrams.

library(dplyr)
library(tidytext)
library(janeaustenr)
library(tidyr) # for separate
library(qdapDictionaries)




#generate unigrams
loss_narrative_by_file_1 <- loss_narrative_by_file %>%
  unnest_tokens(unigram, whole_text_proxy, token = "ngrams", n = 1)



# removing punctuation and numbers ( removing punct / digits faster on unigrams than whole text)
loss_narrative_by_file_1$unigram <- gsub("[[:punct:]]|[[:digit:]]|^http:\\/\\/.*|^https:\\/\\/.*","",loss_narrative_by_file_1$unigram)


#retaining non blank unigrams
loss_narrative_by_file_1 <- loss_narrative_by_file_1[unigram!="",]


# retain meaningful words only
loss_narrative_by_file_1 <- loss_narrative_by_file_1[unigram %in% GradyAugmented]


# remving stopwords


unigrams_filtered <- loss_narrative_by_file_1 %>%
  filter(!unigram %in% stop_words$word)



#check coutns 

unigrams_filtered %>% 
  count(unigram,sort=TRUE)

# after cleansing
unigram       n
<chr>     <int>
  1 claim     56247
2 loss      55068
3 policy    50394
4 insured   47512
5 date      36879
6 coverage  32167
7 plaintiff 31358
8 vehicle   26156
9 umbrella  25055
10 company   24479



# tf- idf ###############################################

# concat claim & filename
unigrams_filtered$loss_file <- paste(unigrams_filtered$loss_Loss,unigrams_filtered$filename_Trimmed, sep = "_")


unigram_tf_idf <- unigrams_filtered %>%
  count(loss_file, unigram) %>%
  bind_tf_idf(unigram, loss_file, n) %>%
  arrange(desc(tf_idf))

### validating results of tf-idf shows top 30% words with highest tf-idf score tend to contain the words associated with cause of loss. How can I narrow down further ?enter code here
$\endgroup$
2
  • $\begingroup$ it would be good to see what have you tried in terms of programming effort. This way, ppl who want to help do not end up replicating your previous efforts. I recommend to revise the post and include the code logic in it. Do see this post on how to create a minimal reprex $\endgroup$
    – mnm
    Oct 9, 2020 at 0:27
  • 1
    $\begingroup$ thanks for adding the relevant code. I've removed the down-vote. $\endgroup$
    – mnm
    Oct 9, 2020 at 14:15

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.